Gearman and TheSchwartz, explained!

[azurelunatic]

Gearman and TheSchwartz are two job scheduling engines that LiveJournal and Dreamwidth and everything on the whole collected codebase use, for different tasks as appropriate.


xb95 that could work. and ofc, now I'm wondering if we can't just use theschwartz workesr instead of gearman workers.
xb95 then it's just one level of abstraction
durandal goes to look things up
goesboom bliiiiink.
goesboom (Jazz) Someone someday want to explain things to me as far as this goes?
durandal ok, stupidity setting in, but, considering how much I DONT know about em (cause I am uninformed like that), which one is actually better suited to running the xpost?
exor674 makes a bug for something at least to do in the meantime
xb95 damn kareila is gone
exor674 .. oh wait it IS evaled
exor674 hrm then
xb95 durandal: TheSchwartz, really. it's durable - which means that if it fails, it retries. or if a worker disappears, it retries.
xb95 goesboom: What it means, wrt Gearman/TheSchwartz?
goesboom xb95: Yes!
durandal mark: ok... that being the case, why are gearman in the mix?
xb95 Well, both have pros and cons, right. TheSchwartz is very stable, solid, but much like a rock. It's a lot of overhead (it's heavy) and a lot of setup and I have a lower impression of it overall.
xb95 But Gearman is very fast and basically lets you just say "hey Gearman, do something for me" and it does.
xb95 Of course, if it happens to crash while doing it, your thing goes away.
xb95 Amusingly, the Gearman team is adding persistent queue support which makes it basically equivalent to TheSchwartz...
exor674 xb95: there we go... that's the issue with it not... failing properly
exor674 as soon as Bugsy wakes up
xb95 And even more confusing, 'Gearman' was named because it's an anagram of 'Manager', as it's a job manager system. 'TheSchwartz' was named after Brad's manager, Mike Schwartz, the VP of Engineering at Six Apart.
Bugsy New Dreamwidth Development bug 1635 filed by anall@andreanall.com.
Bugsy Bug http://bugs.dwscoalition.org/show_bug.cgi?id=1635 enhancement, P-, -undecided-, nobody, NEW, crossposter: get_proxy call is not safe if xmlrpc call failed.
goesboom (Jazz) So, each does what?
JD oh it's not just a spaceballs reference?
goesboom (Jazz) I know absolutely nothing about any of this. But I would sorta like to.
durandal ok, so how hard is it to slave gear to a scwartz master?
xb95 JD: It is not a Spaceballs reference. :)
xb95 Or, well, it could be I suppose. But it was named after Mike.
durandal mark, with Brad, you never know
xb95 goesboom: Both of them are ways of doing asyncronous jobs. Which is like, if you call up your friend and say, "hey, write a paper for me" and then you go play video games while he does. Eventually, when the job is done, he gives you the paper and you can use it.
xb95 The difference between the two is, if your friend pops off to China and becomes a monk, in Gearman your paper never gets done. It's gone. But in TheSchwartz, you get a new friend who will make sure the paper gets done
goesboom nods.
xb95 We use it for things that can run asyncronously; we don't care if they get done *now*, we just want them to get done *sometime*. So we can tell the user "your entry is posted!" and then fire off a job to do the xpost.
exor674 also TheSchwartz jobs are more "do this and tell me later", while Gearman jobs are "do this somewhere else"
xb95 yeah
xb95 dre is more accurate
goesboom Ah, alright. That makes enough sense for our little layman's brain. :)
goesboom Thank you.
goesboom (Jazz) bounces.
xb95 durandal: very tru.e
xb95 durandal: It was proably double entendre!
xb95 durandal: and as for how hard, it's not hard to have a schwartz job fire off a gearman job, but I think the usefulness of doing that is low. MIght as well just run schwartz jobs in lots of places.
durandal mark: *nod*


And that concludes this episode of zorkian and exor674 Explain It All!

Comments

[yvi]

xb95 The difference between the two is, if your friend pops off to China and becomes a monk, in Gearman your paper never gets done. It's gone. But in TheSchwartz, you get a new friend who will make sure the paper gets done

best explanation ever.

[azurelunatic]

Isn't it?

[owl]

I take it this was prompted by the cross-posting blip last night? I never seem to be in channel for any fun stuff anymore :(

[cesy]

So email notifications are done by Gearman and that's why they get lost permanently if something breaks? How much would it cost to change that? (Assuming I'm right in understanding that TheSchwartz is better because it retries stuff, but more expensive in terms of processing.)

[azurelunatic]

Yep! Sure was.

Context leading up to that:

[23:07:36] <xb95> does anybody know if all the crospsosts that failed due to the block eventually retried? or did they just disappear?
[23:07:42] -->| durandal (durandal@cpe-72-177-218-131.satx.res.rr.com) has joined #dw
[23:07:57] <durandal> o/
[23:08:27] * Afuna checks
[23:10:19] <exor674> xb95: pretty sure the xmlrpc handler will die in that case, so if the job permafailed, no -- otherwise... they're stil in there?
[23:10:33] <Afuna> my test entry didn't appear on lj
[23:11:04] <Afuna> and nothing in my inbox
[23:11:13] <exor674> Afuna: probably permafailed
[23:11:25] <xb95> I would hope permafail would fire a notification :S
[23:11:31] <xb95> I know the importer does!
[23:12:02] <exor674> well, the other issue is, the xmlrpc call needs to be wrapped in an eval, cause I'm pretty sure it dies in invalid cases
[23:12:26] <durandal> X-to-LJ are broken again?
[23:12:30] <xb95> I was pondering through the latter half of the ballgame, we should make the xposts happen through gearman workers
[23:12:35] <xb95> then, we run one of them on every machine we own
[23:12:59] <xb95> then, if they realize they're banned, they can sit idle for 5 minutes or something and not accept jobs
[23:13:05] <xb95> so the jobs automatically end up on machines that are known good
[23:13:09] <skud> god, why is video editing SO DAMN SLOW?
[23:13:13] <xb95> it also spreads out the IPs so we don't hit LJ as much
[23:13:14] <exor674> xb95: can gearman retry?
[23:13:33] <xb95> exor: we'd have to have a theschwartz job that handles the retries. so like, theschwartz calls out to gearman.
[23:13:44] <xb95> I know it sounds funky lol
[23:13:47] <xb95> but I think it'd work
[23:13:53] <exor674> xb95: it makes sense
[23:14:20] <xb95> ilu dre
[23:15:16] <exor674> xb95: mebbe even make the theschwartz job generic?
[23:15:39] <exor674> so we can use it for other gearman jobs that can handle their own success/failure
[23:16:38] <exor674> (er permanent failures)
[23:16:52] <xb95> that could work. and ofc, now I'm wondering if we can't just use theschwartz workesr instead of gearman workers.

[azurelunatic]

I ... think that's so? See above thread for the context; apparently Mark is thinking about stuff.

Once Gearman gets persistent queue support that may be irrelevant.

Hm.

[owl]

Mmm, I love technical discussions. Mind you, the fact that last night I was having a Facebook conversation that moved from English through pseudocode to actual valid Java might have clued one in to this already :D

[azurelunatic]

Heeeeeeeeeeeeeee.

[janinedog]

Email notifications actually already use TheSchwartz. I can't say why sometimes they get lost permanently though...I'm not that clear on how it all works. :)

[kareila]

This is really useful and probably ought to be distributed more widely, either on one of the dw_dev communities or on the dev wiki.

Most new developers don't even know how to get Gearman & TheSchwartz working on their dreamhacks, much less why they would want to.

[azurelunatic]

I added Gearman and TheSchwartz to the jargon page, and linked back here to the full context.

If you feel it would be useful, feel free to swipe it, especially given that it's just last night's chatter from #dw.

[elainegrey]

Being completely dreamwidth/lj codebase ignorant, i'm just curious whether Schwartz is part of what could be considered an enterprise service bus platform in the code base.

It seems that the DW/LJ systems essentially require all the same characteristics (http://en.wikipedia.org/wiki/Enterprise_Service_Bus#Salient_characteristics) but developed in parallel with the abstract concept of ESB and developed within one codebase.....

*posts, but figures this is my narrow and shallow interest in ESB development leading me see things that aren't there*

[domtheknight]

Oh, I missed that (must have already been asleep). Thanks for posting! Very interesting.

[azurelunatic]

Hmm. I don't know! Try inquiring in a news post? That's one place where there tend to be a lot of devvish types.

[azurelunatic]

You're welcome! It was fascinating to read, and some of the things that come up in IRC have a wider relevance than just the chattering.

[denise]

I wandered into this entry from the Wiki, and you get your answer three months later! The thing with email notifications is that even though they're handled by TheSchwartz, the reason they don't hit your email is one of three things:

* Your email provider isn't accepting any email at all from Dreamwidth at the moment. (Often happens with places like AT&T, Earthlink, SBCGlobal -- those are the big three offenders.)

* Your email provider isn't accepting any email from one of Dreamwidth's IP addresses, but is accepting mail from one of the other ones.

* Your email provider, or one of the providers between us and your inbox, thinks this particular message is spam, somehow, and has silently refused to relay it.

(There's also possibility #4, which is that you mistyped your email address or whatever, but that's something you'd notice more quickly, I'd imagine.)

In all of those situations, the email gets attempted by DW, just not delivered. Think of it like a phone conversation; in most cases, if the person you're calling (your email server) can't take your call at that moment, they'll let it go to voice mail (email queued for later delivery, and DW will try to deliver it again later), but sometimes you call someone and you get a "number out of service" message (all mail blocked), or "all circuits busy, try again later" (one IP blocked), or the phone just rings and ring and voicemail never picks up (one message not relayed because the provider thinks it's spam), and in those cases DW gives up on trying to redeliver the mail.

So to the layperson it looks like DW just isn't sending email notifications, while in reality, DW is sending them, it's just that your email provider is refusing to accept them for some reason or another. It happens fairly regularly due to the volume of email we send and the fact that a lot of spammers use the @dreamwidth.org email addresses as spam-blast targets, and those spam messages are delivered through DW, which makes some overzealous hosts block DW as spammers even though they're not actually generating the spam, just delivering spam that's sent to a forwarding address. (I want to add SpamAssassin or some other form of spamblocking on the forwarding addresses at some point.)