Download / Archive your posts and/or threads?

[winnielinnie]

Is there any tool (even third-party) that allows a user to "download their data" from this forum, in the same light that Google lets you download your data?

What I'm most interested in is essentially creating offline copies of every thread/discussion I've been in, since I value them as a part of history. (Some great conversations and exchanges too, and a few good laughs.)

To do this manually is highly tedious and practically unfeasible.

 

[Ericloewe]

Good question. I suspect there are plugins to enable this and I also suspect that XenForo's structure is stable enough that this should be fairly simple to scrape automatically.

 

[winnielinnie]

If so, that would be amazing. I don't care if it's slow as molasses in which I'd have to wait for weeks for my "archive" to be ready for download. Online conversations are just as valuable to me as photo albums.

*To be clear though, I'd be interested in more than just "my" posts. Because if it's only my very own posts, then it would lack the actual conversation/discussion of the relevant thread itself.

 

[Ericloewe]

No big, just find all posts, deduplicate the threads and scrape the entirety of them. Optionally also list liked posts. Who cares about big-O complexity when you only have 4000 posts to iterate through. Computers are fast.

 

[winnielinnie]

Sounds like scripting (and trial and error) involved! Was hoping that XenForo (or a third-party tool) has a seamless way to do this?

I wouldn't even know where to begin to search for the existence of such a tool.

As it stands now, from this side of the curtain, I can only see a manual/tedious approach. Are there such tools for this out there?

 

[winnielinnie]

After looking around a bit, I found that there are (paid) solutions to offer users the ability to download their content or entire threads as an exported HTML or PDF.

Obviously, I don't expect iXsystems to purchase such an addon just for my sake.


Another alternative would be to:

1. Figure out a way to curate a list of threads via the "Your content" feature
2. From the above list, remove duplicate entries (so you're left with only unique threads)
3. From the above list, figure out a way to curate a list of all "pages" in the thread
4. From the above list, download the pages in bulk

The problem with curating such lists (steps 1 and 3) is that you can only do so manually, one page at a time. You'll notice that the "Your content" section is paginated, with no true understanding of where the "finish line" is.

Step 2 should be fairly easy using uniq.

Step 4 should be fairly easy using httrack or wget.