From Trac into Gitlab for Tor
Gaba
November 20, 2020
Tor has been using Trac until June 2020, when we moved to our self-hosted instance of Gitlab administered by the Tor sysadmin team. We reached some limitations with Trac as well as were concern on some of the plugins we depended on not being mantained. The challenges on doing this migration sooner were and are related to the capacity that we have to adapt a new ticketing system to our needs.
We’re hoping Gitlab will be a good fit because:
- Gitlab will allow us to collect our different engineering tools into a single application: Git repository handling, Wiki, Issue tracking, Code reviews, and project management tooling.
- Gitlab is well-maintained, while Trac plugins are not well maintained and Trac itself hasn’t seen a release for over a year (since 2019).
- Gitlab will allow us to build a more modern approach to handling CI for our different projects. This is going to happen after the ticket and wiki migration.
We spent several months fixing and testing problems on data migration, from formatting issues to where the information of trac goes to live to in Gitlab. We tested the Gitlab instance with a few projects until we jumped into migrating all data from Trac. You can read about the use cases for a bug tracker at Tor in this ticket.
To accomplish the migration, the Gitlab migration group wrote a number of tools to make the migration happen. These tools were split into two parts: one part for fetching all the state from Trac and a number of tool from turning the Trac state into Gitlab issues and wiki content in various ways. The migration group worked with the engineering teams in the organisation on issues such as splitting up Trac “flat ticket namespace” into the different groups and projects that we wanted to have on Gitlab. This allowed the individual teams at Tor to decide the organisation they wanted to have on Gitlab and allow them to build a mapping that fits better with the project model of Gitlab, where each project have an issue tracker, a wiki, and a repository attached.
We specified a specific date for the migration to happen where all of Tor’s engineering teams were asked to find different means of doing their work than using Trac while Trac was put in read-only mode. During this period the migration group worked together on the actual migration, verifying that all data was properly migrated to a point where things looked satisfying, and then finally we announced that Gitlab was what we would move to next. We did the transition period with start on Friday afternoon over the weekend to ensure that only a minimum amount of disruption would be caused by this.
The period after the migration did involve a bit of support handling from the different teams, but we ar amazed at how quickly everybody picked up the new work flows and we believe that Gitlab have made it easier for engineers to make choices around their respective projects now without needing help from the Gitlab admin team.
We are not migrating away from Gitolite and Jenkins just yet. This means those services are still fully operational and their equivalent features in GitLab are not supported (namely Git hosting and CI). Those services might eventually be migrated to GitLab.
The issues and wiki of the “Tor” project are migrated. There are no other projects in Trac.
- Trac wiki: https://gitlab.torproject.org/legacy/trac
- Trac issues: https://gitlab.torproject.org/legacy/trac/-/issues
Trac issues that remain are really legacy issues, others issues have been “moved” to the respective projects. All the tickets that were not moved to their respective projects have been closed in the first week of July 2020. Next year we will permanently shut Trac down and keep it archived in the Wayback machine.
To request a new account you have to fill the form in https://gitlab.onionize.space/ where we get the request and a few of us attend to them. Through the Outreachy Internship we are mentoring an intern that will help improve this application.
To be able to have all issues in one same board we created a main group “tpo” where all our projects live. The structure for the rest of the projects is:
Organization: host our main wiki, which links to documentation for all projects at TPO. It also hold issues that may not be related to any particular project but are organizational on TPO.
TPA: host any project related to the infrastructure administered by TPO
- Gitlab : Any issue or documentation related to running the Gitlab instance.
- Gitlab Lobby: All issues and development of the django app that is deployed on https://gitlab.onionize.space
- Team: Any issue related to administering TP infrastructure
Core: host projects that are related to mantaining little-t tor
- arti, tor specifications, shadow, trunnel, tor socks, fallback scripts, directory authorities, chutney, tor
- Team: Any issue related to processes for the core team and the wiki.
Anti-Censorship: host projects that work on circumventing censorship with Tor
- gettor, pluggable transports, rdsys, bridgedb, censorship analysis, bridgestrap, emma, state of censorship
- Team: Any issue related to processes of the team and the wiki.
Network Health: it has all the projects related to monitoring the Tor network
- doctor, exitmap, torflow, sbws, helper scripts
- Team: Issues related to network health in general but not to specific projects.
Applications: everything at Tor that is a user facing product
- tor browser projects, tor launcher, https everywhere
- Team: issues and wiki related to processes of the team that work on user facing products
Metrics: everything related to collecting, analyzing and visualizing data from the Tor network
- collector, metrics website, onionperf, weather ,utilities, analysis, exit scanner, exonerator
- Team: issues and wiki related to the team
Community: is for all the projects that help people that help Tor.
- l10n, support, outreach, training
- Team: anything about processes for the community
Web: all projects and code related to the websites that the Tor project mantains
- support portal, community, torproject.org main website, donations, styleguide
UX: all projects related to user experience in all the software we develop at the Tor project
- design, research, media
- Team: anything about the people working on UX at Tor
You can read more about Tor’s Gitlab instance in the documentation.