Plans for a New Forum (complete!)

originally posted by Danielle Tilley

Thanks Brian,

Glad to see all of the work you've put into converting this forum.

I am willing to be an early beta tester of the new discourse if you need anyone.

Danielle
Long time lurker

originally posted by Laneth Sffarlenn

Hey Brian,

Thanks for the update, it truly is appreciated. You're doing an absolutely fantastic job - the new site looks amazing! Thanks to all who've been involved :smiley:

With regards to Discourse, if you'd like an alpha tester whose had some exposure/experience with the communities, I'm sure David and I would be willing to give you a hand as needed prior to the soft launch.

It makes sense to not be migrating user accounts - unless you had the ability to see user activity in the back end of the current setup, it would be a nightmare to have to determine whom to move and whom to purge / archive.

I imagine that you'll be leaving a redirect / migration page here where the old forums would be in the event that an old user returning from some distant past via a bookmark can be informed of the forum move?

This is all so very exciting - the spring clean energy that has come as a consequence and unexpected side-effect of this global paradigm shift is helping tidy up the dust and cobwebs that have clung to our regular haunts and processes for too long!

Looking forward to seeing what's coming next!

Cheers,
Laneth

Brian deserves a medal for this! He is a phenomenon, in a long tradition of phenomenal people who've gone to the mat in support.

Andrew - looking at you, and tribute to Jeff, the original Gryphon.

There is more new stuff coming! Look for front page announcements coming up in the near future/and here's fun, not everything is website related.

Hi all,

An update on the forum is long overdue. The forum migration went on the back burner last fall in order to support Janny as she simultaneously readied 5 separate books for the March e-book release while continuing to write Song of the Mysteries.

Now that those priorities are settled and I'm no longer distracted by my annual winter diversions (Puzzle Boat 7 and Advent of Code), I'm back to it! I've completed research into the various features and extension points in the Discourse forum software and will begin customizing the software to our community's needs and page styles this weekend. I'm aiming for a pre-launch beta test for eager testers in the Feb/Mar timeframe (it will be announced here), with a full switchover by May. This is a firm infant-driven deadline!

Discourse is wonderfully modern (and mobile-friendly), so if you are used to any other forums or communities created in the past decade (Reddit, Goodreads, Facebook, etc), you shouldn't have a problem getting acclimated. There are almost TOO many features offered out-of-the-box, so we may start with a more spartan experience and then gradually open up new features as people get comfortable. I still plan to make sure that everyone here gets first dibs on their preferred usernames and has time to get comfortable with the new forum with plenty of hand-holding – I do not want to lose anyone just because things have become different!

Also, the new forum will have a separate area for website-related posts like this one. I'm very aware that people are here to discuss Janny's books first and foremost, not posts about software updates and 0s and 1s. I have these posts in the Song of the Mysteries topic area for now solely because of the instability of this ancient software, so thank you for bearing with me until we have a reliable "geek" channel going forward.

Happy New Year!
BU

originally posted by Maia

Discourse is good, as has been stated. Being able to use markdown in posts would be a wonderful thing for expression, and it doesn't have to be visible in the UI. I would definitely agree that less is more and am 100% in favour of at least temporarily hiding features. Most people here would seem to value intelligent conversation over fake-reward social media mechanisms (another point in favour of using Discourse). I would guess the driving points for visitors would be:

1. finding answers to questions
2. discussing the books
3. hearing from the author
4. upcoming books and events

None of which require fancy features and could be hampered by such.

Two Exceptions:
Since there is a really lovely wiki (thanks for that!) it might be worth the effort to make it so users can use simple [[Wiki_Page]] links in posts. It would also increase visibility and encourage more collaboration on the work. Most quesions could be answered there rather than endlessly repeated in posts. I didn't immediately see a generic plugin for custom markdown tags in posts and I'm an old school LAMP dev so writing one in RoR is not on. I do see an official spoiler tag plugin. Nice and possibly a base for wiki links.

Regular forum features and organization should cover the rest, but that can be explored in the beta when there is something to point at.

Additional Content Requests:
- Paravia map link
- ways to show support for the author and/or her favoured causes (also, it'd be nice if you could buy all short stories with one click)

Migration is hard. Both the task and the fact that people generally do not appreciate change, however necessary. Not a new thing, but in stating it perhaps it will be a tiny bit easier moving forward. It is certainly worthwhile. Thanks to all for the thoughtful posts and work on the upgrades. It is understandable that these updates would be here and in itself underlines the need for change. Plus, some of us actually enjoy the geeky bits!

Welcome here, Maia, and thank you for your thoughts!

The spoiler tag plugin is already installed on the new forum. No one will ever have to type SPOILER SPOILER SPOILER SPOILER at the beginning of their post ever again. :smiley:

As of last night, the migration of Chat Area data into the new forum is complete. We could use additional sets of eyes on the migrated topics for quality control, if anyone has too much free time.

How can I help?

  • Compare topics against the old copies on this Chat Area and report problems that might have been introduced in transit (consistently funky formatting, many topics in the wrong category, etc.)
  • You can also report minor problems (one reply with weird characters in it, a title that no longer makes sense in the new categorization scheme) but these are less important.
  • This is purely a data migration check! Don't worry about styles or user experience yet.
  • You can report problems directly to me by email at brian AT urizone DOT net, so we don't clutter up the Chat Area and people's email digests.

What should I know before starting?
  • As part of the data migration, all topics were recategorized to the most logical location. The new forum is arranged around categories (the overarching series) and tags (one or more book titles). Instead of navigating 11 separate sub-forums, one for each Volume, you simply create a topic in the Wars of Light and Shadows category and tag your topic with the book you're discussing. This helps reduce visitor indecision (where do I post?!?), opens up discussions to more passing eyes, and allows people to filter on just the tags they care about.
  • Several topics from the early 2000s are missing their first few replies (probably due to ancient bugs in the Discus archiving bot). I opted to retain these topics in case there's valuable knowledge in what remains. You do not need to report these topics if their beginning is ALSO missing in the original Chat Area topic.
  • All topics can be viewed without login, so account registration is disabled. Since the database might need to rollback, I don't want people to have to create their accounts more than once.
  • The new forum may experience unannounced downtimes of up to 15 minutes in the event of a database rollback.

Having read this far, here is the link to the new forum!
https://forum.paravia.com/

What happens next?
  • I will correct big, sweeping problems by rolling back the database and remigrating everything.
  • I will correct minor problems by hand through the new forum interface.
  • Once the data is fairly settled, we will move on to the next phase (probably in the next 1-2 weeks).

What are the other phases?
  • A beta test (2 - 3 weeks), where people can claim their favorite usernames and try out the forum. Emphasis on: improving user experience, visual style tweaks, and identifying the confusing parts so I can write a How-To Guide that helps other people make the transition.
  • A soft launch (1 - 2 weeks), where we stop posting in the Chat Area. The new forum becomes the authoritative one but we don't announce it beyond the regular visitors here until we're sure that all of the kinks are worked out.
  • A hard launch, where we archive and delete the Chat Area, redirect all links and bookmarks, and live in the future!

What are the technical details of the data migration?
669 topics containing over 16,000 replies made the cut. Automation was key to making this possible.
  • The starting Chat Area format was a Discus flat file. Each time someone made a new topic in the Chat Area, Discus created a new HTML file containing the topic, its replies, and all of the HTML/JavaScript needed to render the UI. I started with a one-click task that downloads thousands of files from the webserver and makes copies in a staging area.
  • Next, I wrote Java code that digs into each file, extracts the important parts (title, poster, text, date) and discards the clutter. This cleansing process also eliminates ancient links that return a 404, updates Discus emojis to Markdown emojis, chastises that one reply that had BLINK tags in it, hides any email address posted in the past, and then assigns new categories and tags.
  • Next, I wrote Java code that uses the Discourse API (via a REST call) to pull the clean data in. I can run it all at once or by category. Much of this work was already done in open-source software, so it was a matter of adapting rather than starting from scratch.
  • I ran the import one category at a time at first, since some obvious problems required newly cleansed data. For a few days, it was a continuous cycle of "clean, import, find a problem, rollback, repeat". Most of this cycle takes seconds to run. The import step takes less than a second per topic/reply, but requires a 1-minute cooldown every 200 posts to avoid the API throttle limit – so a full import takes about 3 - 4 hours.
  • I scaled the forum server up to an AWS t3.small as a result of the import because t3.micro (the bare minimum) just wasn't performant. This increases operating costs by about $4.50 per month.
  • While people are vetting the data migration, I'll be enhancing the import step to support "just new replies". This will be used before switchover to quickly catch any new posts without having to reimport everything.

2 Likes

Wow, Brian.

People, this is stunning dedication - give Brian a round of applause and thank him by Using the New Forum!!!

I am into set 11, so Song of the Mysteries is moving along! Put a whole new spin on everything, this book.

I truly hope there will be lots to discuss!!

1 Like

I have created a "Getting Started Guide" to help people get acclimated to the new forum:
https://forum.paravia.com/t/getting-started-guide/707

During the upcoming beta test, I will revise the guide to include other features or idiosyncracies that people find confusing.

Regards,
BU

1 Like

The data migration to the new forum feels solid, so we're now ready for the beta testing phase to begin!

The new forum is at:
https://forum.paravia.com/

The Getting Started Guide is at:
https://forum.paravia.com/t/getting-started-guide/707

What can I do?

  • Create a new account via the "Sign Up" button on the forum homepage using a valid email address. Pick your favorite username, as this account will carry over when we go live.
  • Explore the new forum and get comfortable with the basics: categories and tags, reading and navigating posts, and creating topics and replies. If you create new test topics using the "Sandbox" tag in the Meta category, I'll make sure to delete those before we go live.
  • Post any feedback about usability and style in the "Plans for a New Forum" thread on the new forum. I'm especially interested to hear if there are aspects that make the new forum difficult for your personal use or things you do in the Chat Area that you can't figure out how to do on the new forum. I will do what I can to correct the most urgent issues before we go live, and schedule other requests in the future.

What happens to the Chat Area during the beta test?
  • The Chat Area remains the authoritative place to get news and updates. Janny will continue posting news and updates in the Chat Area during the beta test.
  • Janny, herself, will not be reading the new forum until we switch over. You should continue to post in the Chat Area if you want her to see your post.
  • I will frequently sync new posts from the Chat Area to the new forum so nothing will be lost. This means that you could ostensibly switch over to the new forum early if you wanted to, and you wouldn't miss any news.

What happens to data in the new forum after the beta test?
  • There will be no more database rollbacks. Any accounts you create, preferences you save, or posts you make will carry over when we go live.
  • I will delete any posts with the "Sandbox" tag in the Meta category when we go live.
  • I will not be migrating any new posts back to the Chat Area.

How long is the beta test?
The length depends on how much feedback I receive, and how long it will take to address the critical issues. I'm envisioning 1 week if everyone takes to the new forum like ducks to water, and up to 3 weeks if we need more refinement.

What are the other phases?
  • A soft launch (1 - 2 weeks), where we stop posting in the Chat Area. The new forum becomes the authoritative one but we don’t announce it beyond the regular visitors here until we’re sure that all of the kinks are worked out.
  • A hard launch, where we archive and delete the Chat Area, redirect all links and bookmarks, and live in the future!

Please say something if anything is scary or unclear. I'm probably providing way more detail than necessary in these nerd posts, but I want to make sure that every step we take towards a new permanent home is well-telegraphed, and that the place we end up is welcoming, active, and inclusive.

Regards,
BU

2 Likes

first p0st!!1!

This is beautiful! Thank you thank you for this, Brian!!! I’ve been exploring for a few days now. No obvious issues with the data import, though there is a ton of previous content so we’ll have to wait for others to chime in. The basic functions that existed previously seem to work, and some (like search) are much better. So I’d say it’s off to a great start already.

One point on navigation - the image link to return to the main page of the form might not be not obvious enough on its own. People generally like a clearly worded button or link to click. I found myself appending to the domain a lot to navigate, and most won’t do that.

Hehe. I can confirm it works very nicely on Raspberry Pi and Chrome, which the old forum did not always do. Also very nice to have a place to put geek stuff that won’t get mailed to people who are uninterested!

Thank you again. It’s easy to look at the surface result and miss all the effort that goes into something like this. We’re lucky to have you, and your care for the project shows in the posts and details.

Thanks for the kinds words!

I’ll see what I can do about making the “back to category” link more obvious. On short posts the “Janny Wurts - Official Forum” logo seems reasonable enough, but I agree that the triskelion next to the topic title on longer threads is not super obvious. At a minimum, I’ll update the New User Guide to mention this.

Snagged my user name!!

@Maia An update on your requests:

Easy linking to Paravia Wiki
Adding this one to the backlog of "nice to have"s. There are no existing Discourse plugins that support this but, as you said, using the Spoiler Tag plugin as a starting point should allow me to create a custom plugin. I’ll come back to this as time permits but it should be an easy add.

Easy linking to Interactive Map
I have some modernizations planned for the Interactive Map (think zoom levels and drag-to-navigate). As a result, the current ability to link static pages of the map will be replaced by a more flexible way to link to any “area” of the map. So, I’ll come back to this request once the map has been modernized (sometime later in the year).

Supporting Author’s Favoured Causes
As has been done in the past, this will be handled with ad hoc authorial posts in the Community category. If we end up needing a more rigid platform to handle things, we can revisit.

Buy all e-books with one click
Linking to the page listing all 6 stories will probably be as close as we get to this. Some buyers already have some of the stories, which would require them to remove some e-books from their cart (increases chance of mistaken purchases).

Thanks for your suggestions!
BU

@auricle yer awesome, 'ya gotta know.

This place will undoubtedly evolve and show what it does or doesn’t need, so zero complaints from this quarter on work that may not repay itself. Perhaps we could get enough people behind a subscription Patreon type system. Support the author, download access, input on what we’d like to see (just input - creativity isn’t a machine, much as publishers and readers might like it to work that way). Another idea which may or may not show value in the future. Things here seem to work nicely with just community discussion.

It’d be great to see the map get updated. I love that thing. I’ve had luck with SVG layers and PHP for mapping, if you’re interested in poking into that.

So very excited for this new forum to go live!

Thanks yet again for your efforts. With the new forum, all the key pieces are there. I’ve said before but it’s worth saying again: the Wiki is awesome.

It is a bit difficult to find all the great resources available - the best place is the About the Wars of Light and Shadow category post (thanks!). Maybe stick those on the main Links page under a new WoLaS section? Or link from there to a Wiki page? I had to stumble upon some of those from discus posts and web searches. I can’t say what most visitors to the site are looking for, but I imagine those resources would get enough clicks to justify them a more prominent place.

Which reminds me - the online dictionary is missing terms. Not sure how much trouble that is worth, as the information can be found easily in the books.

Again, no complaints from here. Just one perspective - hopefully it’s helpful input.

We're ready to move over to the new forum any time now. No major problems have been discovered by early adopters this past week and Janny has confirmed that the new forum works on her unique computer setup.

I want to give stragglers a little more time to learn about this switch, since I know that some visitors come here on a schedule of weeks rather than hours. With that in mind, I will plan to do a full switchover on Saturday morning (EDT), February 6, 2021.

What do I need to do?
You do not need to do anything to continue reading the new forum. If you wish to post, you will need to create a new account. This can be done at anytime – in the coming week or even after we have permanently moved.

What happens during the move?

  • I will do one final sync of any new Chat Area posts to make sure they end up in the new forum.
  • All visible links on Janny's main sites will be updated to point to the new forum.
  • Anyone that has a bookmark here will get redirected to a placeholder page explaining that we've moved. This will remain for a couple months. After that, people will get sent directly to the new forum without any explanation.
  • There will be no downtime.

What happens after the move?
  • The new forum will become the authoritative place to talk to other fans and get writing updates.
  • As soon as it's clear that we won't need to ever go back to the Chat Area, I'll permanently delete it and all of the account data. This includes names, emails, profile bios, and one-way hashed passwords.

How do I locate posts from the Chat Area going forward?
  • All posts can be searched for using the new forum's magnifying glass icon in the header. The search is much better than the one here so it shouldn't be hard to track things down.
  • All links in the Paravia Wiki that referenced Chat Area posts have already been updated to point to the new copies in the new forum.

The new forum is at:
https://forum.paravia.com/

The Getting Started Guide is at:
https://forum.paravia.com/t/getting-started-guide/707

Regards,
BU

The move is complete! The Chat Area software has been laid to rest after almost 22 years of service (August 22, 1999 - February 6, 2021).

Enjoy the new digs – there’s telir brandy in the cupboard.

2 Likes

It’s a bid sad to see the old Chat go, but I think we had the longest running chat left in the world on that old piece of software. This new forum’s much more usable and Brian’s done an excellent job getting it up and running and all the old posts migrated across.

1 Like

Is there a way to sort the posts with the latest at the top?

Hi David,

Above the list of categories on the first page are three options:

  • Categories shows the main categories on the left and the topic threads with the most recent replies on the right.
  • Latest shows just the topic threads with the most recent replies.
  • Top shows the topic threads with the most views, regardless of recent activity.

If you always want to see just the latest posts, you can set this in your Profile Preferences. Click on your Avatar, then Profile icon, then Preferences. On the left side of the Preferences screen, find Interface. Here, you can set your Default Home Page to Latest.