Forum:SMW is way faster than info pages

Forums: Index > Watercooler > SMW is way faster than info pages

To give you an idea of what makes things slow, if you use your browser to look at the source for a page (Firefox: "View", "Page Source" or "Ctrl" + u; IE has the same thing), then search for "NewPP" you will find some interesting numbers that measure how hard a page makes the wikia servers work. The last one is most important: expensive functions. The maximum value is 500 but oftentimes the engine will let you go 50 over.

I have been noticing that some of our high value articles have been slow, and so I took a look at reducing these numbers. Some preliminary results below. SMW children is the new {{showfacts children}} templates that will replace {{showinfo children}}. On articles with large numbers of children, this consumes a lot of CPU time. Also as can be seen, the Descendants calculation was also consuming a great deal of processing time, so I converted it to use only SMW functions.

Article	original	after smw children upgrade	after smw descendants upgrade
Charlemagne (747-814)	546	279	121
Charles II of England (1630-1685)	399	274	174

Here is a breakdown of where all the processing is going- counting backwards.

After Smw children upgrade*	274
After Smw descendants*	174	Old descendants processing cost:	100
After removing infobox	148	Infobox cost	26
After removing autocats from info categories*	87	autocats cost	61
After removing facts from info	46	cost facts from info	41
After removing Inbreeding test	25	inbreeding cost	19

Starred items are permanent savings, and these costs were fairly constant whether the article was big or small. -~ Phlox 00:07, 13 June 2009 (UTC)

Good. Not everyone has 20 children, but it must be good even for fewer children. You will be sure to tell us when SMW is really ready to replace info pages? — Robin Patterson (Talk) 00:46, 13 June 2009 (UTC)

There were some dramatic savings on some of the very busy kings, but the huge huge savings are actually for normal articles. Just to give an idea, a fairly rich article at wikipedia like wikipedia:New York City has a total expensive parser count of 38. At familypedia, even a simple article like Agnes von Gutschmid (1842-1924) that Fred just uploaded using standard info pages costed: 108. After switching his article to use some of the SMW templates completed thus far, the cost came back down to 57. The cost would be 21 were it not for the "facts from info template" that is temporary scaffolding allowing both systems to co-exist. So we are looking at a factor of 5 speed improvement just for minimal articles. That means our cache refresh is 5 times faster, and we shall tend to experience less of the prolonged delays on loading or saving articles.

To every Familypedian: the moral of the story is to remember to look at the NewPP number every so often. If you see a figure up around 200, something is very wrong. Start chopping out templates, then do a preview, then display the html source and check the NewPP number. Eventually you will stumble across the stinker(s) that are sucking up all the time. If you see a big number like 250 and can't figure out where the time is going, post a help desk forum note, or tell an admin.

Robin, it would be very hard to forget to let you know when the smw stuff is ready. Those two Maori chaps you sent over are very friendly and all, but I note they become somewhat irritable when my typing slows down... -~ Phlox 01:55, 13 June 2009 (UTC)

Oh my goodness. I just got the handle on info pages and we are changing again? Wonder how long this one will take me.  :-) - William Allen Shade 21:54, 13 June 2009 (UTC)

Will, as I understand it, SMW will be a similar process but easier to use. Individual facts will go into separate boxes on a "form" instead of separate lines of a big edit box, so we will be able to "Tab" to the next one instead of having to aim and click or find the "down" arrow. And the facts are listed at the bottom of a page (as you may have noticed already on many pages, though I hope the order will improve) instead of on a separate page. — Robin Patterson (Talk) 02:37, 14 June 2009 (UTC)

What I and other "followers" would like is an assurance that the info put into info pages will not be wasted but can be converted to SMW facts etc by a bot or similar automatic process. — Robin Patterson (Talk) 02:37, 14 June 2009 (UTC)

Nothing will be wasted, and info pages have given us a huge head start because the data is directly transferable. I would not have undertaken this if any of the data would be lost. So rest assured that everything will be transferred over from info pages. I also hope to be able to upgrade most of the person infobox pages as well via bot, but no promises there. Mind you, some of the info page values have been used in odd ways, such as place names in date fields. People are going to have to clean up those sorts of situations on their own. People will see this sort of mixed up data just as before, but they will really want to correct these problems to get the most out of the SMW features.

William, I think you will see/find that form entry is vastly superior to the error-prone info pages method. And the volume and detail of information one can enter has been increased as well. What is particularly attractive is that when you are entering place names, the Familypedia will autocomplete the entry for you. So if a county name is in 3 different states, you will be made aware of that as you are typing in. This autocompletion is also active for all file and article fields, so there is no problem with typos as with info pages. Some of this stuff is working and you can create your own articles using Form:Person. People are welcome to try creating their own articles using it. Understand that it is under construction and it is not ready for prime time yet. Rtol has been getting a lot of good things done using it, but occasionally I have run over his foot with the bulldozer. He's wearing steel toe boots now.

Certainly if anyone would like to stay with the info pages, they are welcome to. The SMW can coexist perfectly well with info pages. Of course, contributors making that choice will not be able to access any of the new features such as a rich set of database queries, and dynamic categories. For example, one could have a table on one's Userpage that dynamically displays all changes to articles for all surnames the contributor is interested in since the last time the contributor logged in.

I think everyone will be pleased with the features that SMW will deliver for Familypedia. -~ Phlox 04:45, 14 June 2009 (UTC)

Changes are fine as long as we are downward compatible, which I understand we are. I do not like working with forms, though. I'd much rather copy a template from a relative and change what needs changing -- because a lot of information is repeated between relatives. It would be brilliant if the forms were smart enough to recognize that if you start a page by clicking on a red child, at least one of the parents is known. Ditto for red parents and spouses. rtol 07:43, 14 June 2009 (UTC)

As you probably know, everything you do in the form can be done in a normal edit page manner using the various templates {{set birth}}, {{set death}} and so on. Use of the forms has improved, but I have not attended to the workflow issues you remarked on. It is trivial to add a "Copy from wife/ father/ sibling check box as appropriate. The initial thought was to have a one time copy. However, I have been considering whether we should allow persistent inheritance, so that the value will always copy. If the inheritance is understood by the bulk of the contributors, I have no trouble with it. In a form, it would be obvious, because the inherited fields would be greyed/ uneditable while the checkbox is set. But if the bulk of contributors prefer direct editing like you do, it's less clear cut. For folks directly using the templates, it would not at all be obvious to them why values that they set in in the template were not being reflected in the article. The answer is to put error checking in all the templates, but considering the number of cases that would be a pain in the neck. Basically every single event would have to have this code at least for locations, but probably also for fields like participants ("involved people" fields).I only have so much time to devote to this project.

As a footnote to the number of cases for error checks, I should note that there are a whole lot of other life events I haven't created forms for yet. Basically I want to do all the events spec'd in Gedcom 6/ Gramps' XML. Military is a big one because there is good documentation on those- eg enlistments, actions, large communities of war story military guys interested in microhistory of their units. -~ Phlox 19:10, 14 June 2009 (UTC)

I'd say leave the template people to their own devices.

You probably do not need to create a separate form depending on the direction you're working. If I start a page with title "Richard Tol (1969-)" then you can run a query on parents, spouses and children and put these in as suggestions that can be left as is or edited. Does mean breaking the form into two, with a slight delay in between, but it would greatly accelerate data entry. rtol 20:37, 14 June 2009 (UTC)

Well, there are going to be subforms- one per template. Actually they already exist but are out of sync. The way it worked (until I temporarily disabled it this morning) was that when you clicked on say birth label in the {{showfacts person}} infobox, it would allow you to edit just the birth information. It's intuitive this way. A byproduct is that folks will be able to load forms faster but that is not the primary motivation. As for your red text preload idea, the author of semantic forms had the same idea and provided a mechanism for doing this sort of thing. When I read the details I thought we'd like to do that, but I have regarded it as a feature not in the core requirements for first release. I might add it at any time though because- there's no doubt about it- it would be wizzy. The tricky part is it to integrate it with the query that returns the pagename. It's part of the feature that populates a table with localized names- eg mapping "The Hague" to "Den Haag" for an nl template. The red text portion is a 4 hour thing. If lots of folks want the red text or the localized placename/personame thing right away, I'll write it or you are welcome to have a go- take a look at mw:Extension:Semantic Forms#Preloading data with special attention to instructions regarding template-name[field-name]=field-value. The localized place/ person name portion is non trivial though. -~ Phlox 23:34, 14 June 2009 (UTC)