Forum:SMW on Familypedia

Forums: Index > Help desk > SMW on Familypedia

Hey all, here is a conversation I had with Robin tonight and something I would like the community as a whole to be brought up to speed on:

Hello Robin,

There's an excellent reason why your pages weren't displaying on Familypedia for a few hours and that's because we had to turn SMW off there for about a period of three hours. We've been trying to work out a patch because Familypedia was causing massive load on our Apache and MySQL servers. The patch finally took and it was implemented immediately. Pages should be back to normal now.

Robin, we're trying very hard to continue to facilitate your excellent wiki both because we do sincerely want to learn about and improve Semantic MediaWiki so other sites can be a storehouse of information like yours. But the template system you've implemented is insane (I mean that more in the good way than the crude way) and all of those queries and conditions and parameters that you get to handle of this information is simply overwhelming our DBs more and more as more and more content gets added. Frankly, if Familypedia were on its own server, it would have overwhelmed that stand-alone DB server a long time ago. It's a miracle our system has managed to handle this this long.

In other words, there are simply too many blank pages. Too many expensive queries. Too heavily-coded templates. We're going to continue working on this on our end, but this is something you're going to /have/ to address on your end. Those templates need major reworking, even if it means cutting down on some of the functionality, in order for this to continue to work.

--daNASCAT (Help Forum) (blog) 04:47, November 13, 2010 (UTC)

It would be interesting to have some expert attention to what is generating the server load. What does "there are simply too many blank pages" mean?

In terms of our templates, is the problem in our "core" templates for displaying a person page, which queries the dates for the children, or is the problem the computation of "ancestry" and "descendants", where changing one page can effect thousands of other pages? It would be nice to have some instrumentation and logging to find out where the problem lies. Thurstan 07:17, November 13, 2010 (UTC

I would "think" too many blank pages refers to Special:ShortPages, MANY of those could be merged into a parent page to reduce the number of pages the server has to index and work with. I really don't know what is going on with the need for so many separate Translations pages... Also it would probably help with the server load if the templates only asked for each piece of information once on a page. This could be done by storing the query in a variable either using the #arraymap function or something else such as the variable extension. SimAnt 18:49, November 19, 2010 (UTC)

I don't "think" that "blank" is the same as "small", so I don't think that is the answer. These "small" pages are small because they only contain one template: they display a lot of text, so they cannot easily be merged back into their parent. We are still waiting for elucidation of what the new rules are, after the patch, so we have some idea how to fix all the pages which have been broken by the patch. Thurstan 20:45, November 19, 2010 (UTC)

Only one query asking for the same thing on the same page. other formats such as format=count aren't restricted by this though. SimAnt 04:25, November 23, 2010 (UTC)

Thank you for that: none of us had managed to guess it. Looks like we need something like the "variables" extension to cache attributes (like "sex") that are used several time on a page. Thurstan 19:56, November 23, 2010 (UTC)

I don't think it is working like that, which is causing a lot of our problems:

• {{#show: Edward Cook (1869-1919)|?birth_date}} returns "21 August 1869"
• {{#show: Edward Cook (1869-1919)|?death_date}} returns "7 August 1919"

so the second query for a different property of the same page is failing. Thurstan 20:58, November 25, 2010 (UTC)

Yah, i noticed that two about #show and the exact same page being searched. I'm not sure what to do with that, I sent a Special:contact yesterday, since this is probably an oversight. SimAnt 21:36, November 25, 2010 (UTC)

Problems continue

When I try to display pages I get boxes with templates in them, like #show: See for example Joseph Doty (1651-c1732) This happens on every person page I look at, and no part of the page displays normally. DennisDoty 16:04, November 13, 2010 (UTC)

Confirmed Joseph Doty (1651-c1732) as Dennis described - same as what all of them looked like yesterday. But a move back to normality is to "Edit with form" then just "Save". Page definitely recognizable; however, many box labels are links, and other aspects of the page are wrong (or maybe missing - I don't know what else is meant to be on that page). — Robin Patterson (Talk) 01:42, November 14, 2010 (UTC)

Patch

I cut the number of generations in the ancestor and descendant lists on the /sensor pages, and the number of generations on the /descendant pages.

As I have noted before, the trouble with the servers started when Thurstan began creating a large amount of /tree pages. rtol 14:26, November 16, 2010 (UTC)

Simplification

Note that I've started simplifying the main templates -- particularly, I'm removing the multilinguality and the links to forms. This improves the display, and makes it easier to read (and fix?) the templates.

Note also that I have very little time for this. rtol 22:11, November 25, 2010 (UTC)

Thank you for what you manage to do, rtol. If we remove that part of our multilinguality, we should try to find a replacement. Phlox said he based ours on Commons:. — Robin Patterson (Talk) 22:27, November 26, 2010 (UTC)

Removing some of the multilinguality has not restored the standard input form. Please see http://familypedia.wikia.com/wiki/User_talk:Pdlloyd#Biggest_problem_with_forms for Pamela's picture of the current form and comments on it and my response. Seems we need to replace all instances of {{String}} unless we can redirect them. — Robin Patterson (Talk) 02:48, November 30, 2010 (UTC)

I think that is the story: {{String}} uses an SMW query of the same base page Strings:en, so it falls foul of the patch. Thurstan 03:18, November 30, 2010 (UTC)

Are we waiting on wikia or on ourselves?

It doesn't seem that there's been progress in a while - form-derived pages still don't display properly, and the (English) labels don't appear on the form page. Are we still waiting for the Wikia staff to fully re-enable SMW, or have they already done so? If the latter, do we know what needs to be done to counteract the side effects of the template modifications that were supposed to reduce server demand? Bruce Kendall 20:50, December 12, 2010 (UTC)

March 2011 another set of problems

User:rtol alerted me to the latest set of problems, as noted on my talk page and on that of User:Thurstan. Then rtol managed to contact Wikia and get a reply, and tells me "I guess that the problem is that the latest version of MediaWiki (installed on March 17) is incompatible with SMW".

A few minutes ago I checked Special:Version. We have MW 1.16.2, SMW 1.5.5, SF 2.0.8, and Semantic Drilldown 0.8.1. Surely the SMW maintainers are aware of the latest incompatibility? Maybe we need the latest SF, which is 2.1.1, and/or the latest SMW, which is "Last Version 1.5.6 (2011-02-24)"

--- Robin Patterson (Talk to me) 07:05, March 19, 2011 (UTC)

Form:Person still performs passably though without most of its labels, and an article I've upgraded using it has no contributors shown despite being published three times: Jacob Arney (1776-1859). --- Robin Patterson (Talk to me) 09:04, March 19, 2011 (UTC)

Notes and citations now seem to get no heading on some pages, and get one of the symbols at the left replaced by something in this form: [[Special:formedit/General info/Sophia Jane Neich (1846-1926)|]]. --- Robin Patterson (Talk to me) 04:36, March 22, 2011 (UTC)

That is because {{string}} is broken again (possible because of the "type" problem), so the translations are broken: there should be a "General" (or equivalent in your language) between the "|" and the "]": without this, it won't display properly. The same problem is killing the button captions on the "/sensor" page. I suggest winding back some more of the translations (as we have partly done on the form). Thurstan 05:13, March 22, 2011 (UTC)

daNasCat is riding to the rescue on the wider problems

perhaps we should do away with our own {{string}} and use MediaWiki's int instead. rtol 13:54, March 22, 2011 (UTC)

Our internationalization is meant to be based on Commons, e.g. Commons:Commons:Language templates, the basis of our Familypedia:Language templates. I think we use a little of it, somewhere: {{de}}, {{en}}, etc, are under Category:Internationalization templates. Maybe Phlox departed too far from Commons. And maybe we should liaise more with TranslateWiki. --- Robin Patterson (Talk to me) 05:31, March 24, 2011 (UTC)

{{string}} is a local, unsupported shell around the global, supported int. The only purpose of {{string}} is to accommodate non-English speaking visitors who are not logged in. That's a small benefit. As {{string}} frequently breaks and messes up other templates, it comes at a high price. Ditch it, I say. rtol 08:29, March 24, 2011 (UTC)

SMW and Redirects

This version of SMW seems to have problems with redirects: for example Hugh de Vermandois (1053-1101)/tree does not show his paternal grandparents. This is because his father, Henry I, King of France (1008-1060), redirects to Henry I of France (1008-1060). If we can't get it fixed (and can't see how to code around it) then we have to change all the references after each "rename", otherwise the trees and Ahnentafels will break. Thurstan 03:54, March 25, 2011 (UTC)

That explains some recent oddities. Descendant pages too, I suspect. Maybe the latest version of SMW will fix it if we get that. Other sites must have noticed it. Anyway, I've added a caution to Help:Redirect. --- Robin Patterson (Talk to me) 10:49, April 1, 2011 (UTC)

Diagnosis

It appears that the display problems we had earlier this month were due to a faulty installation of the new version of MediaWiki. This has been restored.

Many SMW templates still do not work, though. This is because there is a 1000-char limit on many SMW operations. Previously, this was the case for #replace: (which messed up properties ancestors and descendants, and templates showfacts descendants and showfacts tree02) but it now affects additional operators. I've alerted Wikia. rtol 08:57, March 31, 2011 (UTC)

This took a bit longer than hoped. We now have a Familypedia-specific limit. It's 2000 chars. That's enough for {{showfacts descendants}} but not for {{Set order of Charlemagne}} and {{Showfacts set ancestors}} (which drives married cousin and inbreeding). I've asked for more chars. If we can't get that, we'll need to rework quite a few complicated templates. rtol 19:39, April 5, 2011 (UTC)

5000 chars now. Order of Charlemagne is working again. I'll see what I can do with inbreeding. rtol 05:22, April 7, 2011 (UTC)

Good work, Richard. I wonder whether an additional generation on the descendant lists might be of more value than inbreeding. Maybe inbreeding could have some rounding down to zero when it's very small - after all, it's only an average, and it's possible for any man to have inherited absolutely nothing from each grandmother, with 25% being only the commonest percentage. (See http://en.wikipedia.org/wiki/Introduction_to_genetics#Genes_and_inheritance.) Robin Patterson (Talk) 14:17, April 8, 2011 (UTC)

November 2016

We now have Semantic Drilldown (Version 2.0.1) and Semantic MediaWiki (Version 2.4.1) and - listed in a separate table of Special:Version - Semantic Forms (Version 3.7). (incidentally, SF is, this month, having its name changed to Page Forms (see mw:Extension:Page Forms); however, it may be a while before Wikia sites get a more recent version.)

Is that why Form:Person seems to be inaccessible except through a form that asks us to specify which form we want to use? Or are our linkages from infoboxes to Form:Person now going through a different channel? I hope we can get back to the state where we can click on a name in an infobox and get straight to Form:Person.

-- Robin Patterson (Talk) 02:47, November 8, 2016 (UTC)

We can still get to Form:Person for existing pages, because the "Edit facts" link (which I have always ignored) still works. Another thing that still works is Special:FormEdit, with the "Special:FormEdit/Person/Henry_Harvey_(1851-1906)" style syntax. Clicking a red link in the "info person" box still brings up Form:Person, though without the pre-filled values, so it looks like we can go up the tree but not down. Also note, looking at a page like Henry II of England (1133-1189), and looking at the mini-bio and at the child box, that the SMW-gnomes seem to have done something more with (automatically) Julian dates which breaks our code. This suggests to me that {{Showfacts children}} need to be changed to fix the display. Maybe its red links could be replaced with "Special:FormEdit/Person/..." links as well. Thurstan (talk) 03:04, November 8, 2016 (UTC)

Thank you for expertise. We may have to amend our help pages to guide users through the forms menu. Incidentally, Chinon/bdm shows a new superscript "JL" for Julian dates; does that help anyone recoding? -- Robin Patterson (Talk) 02:12, November 13, 2016 (UTC)

A refresh of the Category:Facts articles- person seems to have restored the "default form" setting, and so the "Edit with form" button now appears. Thurstan (talk) 00:35, November 14, 2016 (UTC)

Later the "Edit" button is the standard, with "Edit with form" on the drop-down menu. Satisfactory if not ideal. -- Robin Patterson (Talk) 05:50, December 26, 2016 (UTC)

On another note, based on one test, has this update fixed the page-move bug? Thurstan (talk) 06:10, November 13, 2016 (UTC)

Seems that something has. -- Robin Patterson (Talk) 05:50, December 26, 2016 (UTC)