Doing It Right: hacking out a spec for a native DW twitter importer

[azurelunatic]

This is what I have so far; of course it's drafty (sketchy? something?) as all hell, but I figured I should have it out of my brain and onto bits that others can see and chatter about. Going to run through other places too.

Twitter-importer for DW, a spec [draft].

Must authenticate to Twitter properly.

Must collect all tweets, whether from a locked or unlocked account. (Twitter account being locked should not entirely prevent from public posting, because some people are merely locked to keep out the spambots.) Should collect retweets too, if possible the whole thing [oldschool misses the trailing characters].

Should be able to set a security for all Twitter entries that is distinct from the minimum security; some people prefer to post their Twitter imports to a custom security group even if their minimum is public. (Locked twitter account and locked entries would maintain security for people who need it.)

Must be able to have timestamps or no timestamps. Must be able to have linkback to original tweet, or no link back. (would be nice to have optional links for in-reply-to, location, and other doodads, but those seem like they'd be harder to get.)

Should be able to set tags (suggest "twitter"). (This will help with paid users' filtering.)

Should be able to set userpic (if they have a userpic called "twitter", maybe have it pre-selected for them?)

Mine counts the day's tweets. I like that. Lets you know what you're in for.

Should be able to customize the subject, with variables to insert the date if you want to, in your exact preferred configuration.

Should be able to cut tweets, either all of them entirely, or after some small reasonable number. My service does 5; LJ's does 10. Either of these are reasonable.

Should be able to customize the cut text.

Should be able to customize the posting time in a way that does not kill the server. Exact time seems like something that too many people would try for the same time, and things would bog down. Though a reasonable effort should be made to a) have the things post at the same time for the same person, and b) get the tweets of a particular span, most popular likely to be the whole day's twittery.

Should be robust enough to try again if Twitter failwhales for any reason while attempting to fetch.

Should be able to choose whether to import all tweets, or only tweets that lack an in-reply-to. (without in-reply-to is good for not having half-a-conversation, but bad if what you wanted to do was archive it all for your own use).

Might want to import from more than one Twitter account? (paid feature)

Should expand as many as possible shortened URLs, to prevent the failure of a shortening site from making your old links useless.


[separate-but-related, having Twitter taking over the Latest page would not do at all; if it ever threatens to dominate, it should be split out or something.]

Comments

[senmut]

You are a beautiful person for applying brain power to this.

I like my import, but I also do understand those who do not wish to see my twittered day.

[azurelunatic]

I've had the advantage of trial and error and what I like in my own: wibbble kindly developed and hosts the script I use, and I've been asking off and on whether something or other were possible, and often it has been. So mine is tweaked almost exactly to my ideals. It's lovely. So I have that to compare to some of the other implementations (LoudTwitter, Twittinesis, LJ's, and goodness knows there are probably others), and what people complain about, what works for me, what doesn't, and why.

[wibbble]

Getting the 'in reply to' and location links isn't difficult at all - the required data is in all the stuff returned by the API calls.

I actually recommend giving their API docs (http://dev.twitter.com/doc) a quick look over to see what you can/can't do - it's fairly readable and the API itself is quite nice to work with. If there's anything that's not clear, gimme a shout - I'll be around on IM tomorrow. (I might still be showing as available - that'll be the iMac upstairs. :o) )

[azurelunatic]

At the moment, I'm slightly zombified and hammering on my CV, but yes, I should definitely take a look at it, and now I've got it open in a tab so I will. Eventually. ;)

[geekgirl]

My brain aches just reading this. Need to get my head out of the Drupal books soon and back into other things.

[automaticdoor]

This would be my dream, omg.

[jeshyr]

Looks utterly brilliant!

One thing you missed: option to turn #hashtags @names in tweets into appropriate links to twitter hashtag searches/username pages.

[jeshyr]

If I could automatically set my tweets to post privately I'd probably start importing them for archive purposes!

Can I get authenticated RSS feeds of everything-except-tag-x from Dreamwidth though?

[azurelunatic]

Oh, yes. So I did. And a little bug that we ran into when wibbble was implementing mine, so I'll include it here: must be able to link multiple #hashtags and @names and expand multiple links. Because sometimes it's #omg #wtf #bees time.

[exor674]

I'm assuming that this is going to be scheduled, not "ticky clicky run once" like the LJ importer?

( Maybe I should see if I can bribe whoever does the scheduled posts implementation to make that more general for other scheduled tasks )

[azurelunatic]

Yes, this would be run-once-every-24-hours.

[andy]

It is scheduled on LJ as well; it posts on noon in the user's timezone, without allowing to tweak it.

[wibbble]

Doing a one-off Twitter 'import' isn't viable - they don't make really old stuff available. :o(

[azurelunatic]

In going through the docs, I found that we could only suck up to 3,200 items anyway if we were trying to go for an import-whole-thing. So while it might be nice to *try* for an import-whole-thing, it would only work for people with ~3,200 or fewer updates, and even that does not look guaranteed. (REST API for 3,200; the others have limits of 800.)

[matgb]

Should expand as many as possible shortened URLs, to prevent the failure of a shortening site from making your old links useless.

Yes, this. I know this is an issue, but my brain doesn't always process it--I currently use Delicious, but if that goes the best workaround I can create is Twitter, unless Memories are vialbe with the new work being done on them.

I'd like to be able to not import tweets that start with the @ symbol, if I want atweet read by many I reword it to put the name further down.

[wibbble]

Most Twitter clients, including their website, set the 'in_reply_to_status_id' option for replies, which is how you're 'supposed' to decide if a tweet is a reply or not.

I don't think it records that option if the tweet doesn't start with '@', but I'd need to check.

[matgb]

I'd rather it didn't do that though. I get why it does, but I've found that a bit buggy, and if I reword a reply to show generally it should.

Plus I sometimes just type @azurelunatic or whoever to ask something, and wouldn't do that as an actual reply.

[jeshyr]

Ditto to that. my client definitely didn't always set that correctly last time I looked.

[jeshyr]

Note about expanding shortened links: Depending on how you do that (checking 301/302 redirects vs using APIs) it would be cool to be able to expand URLs that point to bookmarking services like delicious, pinboard, diigo, etc. These, as we were all recently reminded, may be as transitory as the dammed tinsy URL services :(

[azurelunatic]

wibbble's implementation for me expands Delicious links. The t.co shortened links that Twitter is supposed to be rolling out for everybody might make that easier (especially if they pre-expand obfuscated addresses before re-wrapping them), but no clue about how exactly that's supposed to work.

[wibbble]

Speaking of which, that reminded me to add expansion for t.co links, which should be live from the next run.

[wibbble]

This is the code (in Ruby) I use to expand the URLs:

def expand(url)
  request = Curl::Easy.perform(url) do |curl|
    curl.follow_location = true
  end
  request.last_effective_url
end

This is using the 'curb' gem, which is basically a wrapper around libcurl. That should follow any chain of redirects until it gets to the final source - so as long as the bookmarking/shortening/whatever service just feeds a 301/302 redirect to the browser, that'll get the original source.

I think this is the best behaviour - in a case where you might not want to go all the way back to the original source (say, because there's comments on a social sharing-type site), you'd normally not be posting a link that'll redirect.

One improvement I should make is in URL-detection. At the moment it matches against specific known URL shorteners, but it wouldn't be that difficult to just feed it a generic 'this is a HTTP/HTTPS URL' matcher instead and feed all URLs in tweets through the expander. That might have unintended side-effects on sites that do redirection as part of their normal processes (like every site using the LJ codebase).

[ephemera]

mmmm - that sounds good :D - lots of *choices* about how I want my tweets to show up in my journal, which LJ's tool seems so determined not let us have.

Random additional option that's crossed my mind, but would be complex if not impossible, to have the day's tweets-without-@relies, cut-taged after the first x, followed by the day's tweets-with-@replies, always behind a cut tag no matter the number, (or, I guess, the option to have the 'no conversations' list in one post and the 'complete with conversations' list in another post, with different default filtering)

[azurelunatic]

Oooo.

I like the way you think. I would think that the way to have that easily would be to not disallow duplicate import-accounts set up for the same twitter user, and then to have different settings on each import-account: one pulling only the not-conversations and posting public, and one pulling the whole thing and posting in one's choice of filter.

[azurelunatic]

zarhooie: omg best idea ever
so for twitter imports, if DW does them natively ever, we could scan the import material for hashtags and then tag the entry appropriatgely.


azurelunatic: yes.
what I'd like is a checkbox that says "use hashtags as tags for entry"
in setup
and a link for advanced settings
and one of those advanced settings would be hashtag to tag mapping