How to move from Jira to a self-hosted GitLab: a step-by-step guide

Categories: 
Tiago Correia
1 April 2021
Senior Software Developer – Tiago Correia – explains how to move from Jira to a self-hosted GitLab setup

Senior Software Developer – Tiago Correia – explains how to move from Jira to a self-hosted GitLab setup.How to move from Jira to GitLab

If you just care about the code, you can find it here. The code has some error protection and flags to only perform certain stages, some caching, etc. The explanation below will omit most of this since I want to keep it simple.

A bit of context

At Panintelligence we've used Jira for years now.

It was chosen because it gives us all the flexibility one could ever want: custom fields, workflows, different screens depending on a number of conditions, etc.

Once we adopted Zendesk to track customer requests, it was easy to integrate Jira: allowing links from multiple customer tickets to one Jira issue (if multiple customers were asking for the same thing) or multiple Jira issue to one Zendesk ticket (if a customer was describing different issues on their ticket).

Then we switched our repositories over to a self-hosted GitLab environment on AWS.

GitLab and Jira integration is amazing (even letting you view Jira issues inside Gitlab)!

As the Cloud team got settled, they started using GitLab Issues rather than Jira to track their tasks citing simplicity. I started using GitLab Issues instead of Jira to track the to-do list on the small projects we keep that support our day-to-day activities.

We slowly came to the realisation that perhaps Jira was overkill for what we wanted to do and started looking into how we could use GitLab to do the same thing.

Gitlab Jira
Workflows
Workflow automation
Zendesk Integration 1
Custom fields
Issue manipulation via git commits
Time tracking
Labels
Epics
Multiple assignees
Markdown support
Mermaid-code blocks2
BI Friendly ~3 ~3
  • 1 We thought it did at first but it turned out the integration doesn't support self-hosted GitLab. So I created the integration myself... more on that in the future.
  • 2 Mermaid lets you easily make graphs in markdown without having to make a separate image. It's a nice convenience.
  • 3 Having an overview of our issues in our Panintelligence Dashboard is important for our planning and reporting. Both Jira's and GitLab's databases are not very BI friendly. Jira moreso than GitLab. However, nothing that some data wrangling can't solve.

While we made use of Jira workflows, sometimes the restrictions between the statuses turned out to be a hoop you needed to jump rather than a benefit. In some of the projects we ended up letting you transition from almost any status to any other status. With that in mind, we discarded the importance of workflows.

Custom fields however, we made heavy use of. I thought we should be able to use GitLab's labels for that instead.

Gitlab has two types of labels:

  • Scoped: e.g.: Type::Improvement, Type::Bug, Type::Task / Status::In Progress, Status::Ready to Test
  • Unscoped: e.g.: Scope Creep, Roadmap Item

Scoped labels only allow one label of that scope to be attached to an issue. This could replace our single-selection fields.

Multi-selection fields (such as through which cycles of manual testing an issue went through), would have to be unscoped labels with a prefix, e.g.: Cycle:1, Cycle:2, Cycle:3.

Migration

Step 0: Migrate Jira data using GitLab's tool

GitLab has a native tool to migrate from Jira to GitLab. However, this tool does not migrate any comments and there is no way to migrate the custom field values, as GitLab doesn't have those. Issue types and priority get migrated as plain text in the issue description.

This is all less than ideal, but it's a start.

Both Jira and GitLab have an API, so nothing stops me from doing this myself.

Step 1: Obtaining the Jira data

Since Panintelligence was already reporting on the Jira data, I made use of the dashboard feature that returns a chart as JSON, essentially making a custom API over Jira:

https://{{your.dashboard.domain}}/pi/export/json?chartId={{chartId}}
    

This way, I'm letting our dashboard do some of the data matching and saves me the trouble of processing the data myself. The added bonus is that the Dashboard also reports over the Zendesk data, so I am able to add in the Zendesk issue IDs to the same chart.

Alternatively, Jira's API can be used to get all the issues by calling the URL below:

https://{{your.jira.domain}}/rest/api/2/search?expand=names&jql=
    

This will give you a list of issues and all their metadata.

The one piece of information that is missing, is the comments. For this you can find the comments for each issue by calling Jira's API with:

https://{{your.jira.domain}}/rest/api/2/issue/{{issue code}}/comment
    

The above should give us all the issue metadata and all the comments.

In the next step, I'll show how to obtain each one of those programmatically.

Step 2: Populating GitLab with the Jira metadata

Since GitLab's migration tool adds Jira's issue ID as part of the title, we can use the GitLab search issue endpoint to match the issues together:

https://{{your.gitlab.domain}}/api/v4/groups/2/issues?search="[{{jiraIssueCode}}] "
    

With that we can now start populating GitLab with the following map:

  • Jira Metadata -> GitLab Labels
  • Zendesk Links -> GitLab Labels
  • Jira Sprints -> GitLab Milestones
    • GitLab uses "Iterations" as Sprints, however GitLab API doesn't provide endpoints for Iterations other than GET. The alternative was to use Milestones instead.
  • Jira Comments -> GitLab Comments

The one thing that isn't being migrated are the Jira attachments. This isn't such a big problem for us since most of the attachments were present in Zendesk.

Step 2.0 Get the data from the dashboard

const Dashboard = require('./dashboard.js');
const dashboard = new Dashboard("https://your.dashboard.domain");
dashboard.authenticate("username", "password");
const data = dashboard.getChartAsJson(123456); //this would be the id of the table you made
const rows = data.data;
const columns = data.displayNames;

For contents of require('./dashboard.js'), see full dashboard.js.

The dashboard returns the chart json data with columns and data separated (as seen above), so I want to make each row of data into a json object first.

const jsonRows = rows.map(row => {
    row.reduce((r, value, index) => {
        r[columns[index]] = value;
        return r;
    }, {});
});

Step 2.1 Attaching the comments

Then I get all the jira comments and attach them to each row. This... could take a while.

const Jira = require('./jira.js');
const jira = new Jira("https://your.jira.domain", "your auth token");
const appendJiraComments = async (jsonDoc) => {
    const issueCode = `${jsonDoc["Project Key"]}-${jsonDoc["Issue Number"]}`;
    try {
        const result = await jira.getComments(issueCode);
        jsonDoc.comments = result.comments;
        if (!result.comments) {
            console.warn(`No comments for ${issueCode}`);
            jsonDoc.comments = [];
        }
    } catch (e) {
        console.error(`Failed getting comments for ${issueCode}`);
        console.error(e);
    }
}

for(let i=0; i<jsonRows.length; i++) {
    await appendJiraComments(jsonRows[i]);
}

For contents of require('./jira.js'), see full jira.js.

Then we can add them to each GitLab issue.

const Gitlab = require('./gitlab.js');
const gitlab = new Gitlab("https://your.gitlab.domain", "your private token", "username who you want to impersonate for most things");

for(let i=0; i<jsonRows.length; i++) {
    const issueCode = `${jsonRows[i]["Project Key"]}-${jsonRows[i]["Issue Number"]}`;
    await gitlab.addNotes(issueCode, jsonRows[i].comments);
}

For contents of require('./gitlab.js'), see full gitlab.js.

There are a few steps within addNotes, but the essentials are covered below.

Starting with finding the issues.

GitLab's issue importer always adds [KEY-1234] to the beginning of the issue title, so the request below searches for that.

// const gitlabUrl = ...
const issues = await request.GET(`${gitlabUrl}/api/v4/groups/2/issues?search="[${issueCode}] "`, { "Private-Token": this.privateToken });
const match = issues[0];

After obtaining the right issue, add each comment to the GitLab issue, preserving (if possible) the commenter and original date. This works by using GitLab's impersonation feature by passing a { "Sudo": "username" } header with the request.

To preserve the commenter, I find the match between GitLab and jira users through a key-value pair object (jiraToGitlabUserMap). Anyone not mapped ends up as our bot, henchman:

//...
const authorUsername = jiraToGitlabUserMap[comment.author.name] || "henchman";
//...

If the commenter is henchman then I want to keep a note of who the original author was:

//...
if (authorUsername === "henchman") {
    message = `${comment.author.displayName} said:\n> ${message}`;
}
//...

Finally, the message is added:

//...
// const gitlabUrl = ...
const url = `${gitlabUrl}/api/v4/projects/${match["project_id"]}/issues/${match["iid"]}/notes`
await request.POST(url, headers, {
    body: message,
    "created_at": (comment.created || "").split(".")[0] + "Z" // preserve the date and time of the comment!
});
//...

Note: using created_at is only permitted if the request is made with an admin token.

Put it all together:

const request = require('./request.js');
//...
// const gitlabUrl = ...
for(let i=0; i<jsonRows.length; i++) {
    //...
    // instead of using addNotes:
    jsonRows[i].comments.forEach(async (comment) => {
        // In the repository code, there's some caching going on here so we only get the issue if we've not gotten it yet
        // See  gitlab.findIssue()
        const issues = await request.GET(`${gitlabUrl}/api/v4/groups/2/issues?search="[${issueCode}] "`, { "Private-Token": privateToken });
        const match = issues[0];

        const authorUsername = jiraToGitlabUserMap[comment.author.name] || "henchman";
        let message = comment.body;
        const headers = {
            "Private-Token": privateToken,
            "Content-Type": "application/json",
            "Sudo": authorUsername
        }
        if (authorUsername === "henchman") {
            message = `${comment.author.displayName} said:\n> ${message}`;
        }
        const url = `${gitlabUrl}/api/v4/projects/${match["project_id"]}/issues/${match["iid"]}/notes`
        await request.POST(url, headers, {
            body: message,
            "created_at": (comment.created || "").split(".")[0] + "Z"
        });
    });
}

You'll see there isn't any conversion of Jira's weird markup language to regular markdown. If you've got something for that, let me know!

The links between zendesk and the jira issues come from that dashboard chart.

As previously mentioned, the links between Zendesk and the GitLab Issues are going to be maintained via labels, so I just need a map of JiraIssue -> Zendesk Tickets.

// ...
// const jsonRows = ...
// ...
const linksMap = jsonRows.reduce((acc, row) => {
    const issueCode = `${row["Project Key"]}-${row["Issue Number"]}`;
    if(!Object.keys(acc).includes(issueCode)){
        acc[issueCode] = [];
    }
    acc[issueCode].push(`Zendesk:${row["Zendesk Ticket ID"]}`);
    return acc;
}, {});

For each jira issue that had a link to Zendesk, I should have 0 or more Zendesk:12345 labels.

Then I just need to add those to the related issue:

// ...
// const gitlabUrl = ...
await Object.keys(linksMap).forEach(async (issueCode) => {
    const labels = linksMap[issueCode];
    // Remember that issue cache I mentioned? Yes, this is why gitlab.findIssue is useful.
    const issues = await request.GET(`${gitlabUrl}/api/v4/groups/2/issues?search="[${issueCode}] "`, { "Private-Token": privateToken });
    const match = issues[0];
    const headers = {
        "Private-Token": privateToken,
        "Content-Type": "application/json"
    }
    await request.PUT(
        `${gitlabUrl}/api/v4/projects/${match["project_id"]}/issues/${match["iid"]}`, 
        headers,
        { "add_labels":  labels.join(",") }
    );
});

Step 2.3 The rest of the metadata

Jira contained quite a few fields we don't want to lose track of. Things like issue priority, type, workflow status and test iterations.

Again, I'll make use of that Dashboard table's json data:

// ...
// const jsonRows = ...
// const gitlabUrl = ...
// ...
await jsonRows.forEach(async (row) => {
    const issueCode = `${row["Project Key"]}-${row["Issue Number"]}`;
    const issues = await request.GET(`${gitlabUrl}/api/v4/groups/2/issues?search="[${issueCode}] "`, { "Private-Token": privateToken });
    const match = issues[0];
    const labels = Object.keys(config.jiraToGitlab.fieldsTranslation)
        .map((key) => {
            return valueToLabel(row[key], key);
        })
        .filter(label => !!label); //remove nulls

        const headers = {
            "Private-Token": privateToken,
            "Content-Type": "application/json"
        }
        await request.PUT(
            `${gitlabUrl}/api/v4/projects/${match["project_id"]}/issues/${match["iid"]}`, 
            headers,
            { "add_labels":  labels.join(",") }
        );
});

You'll see a config.jiraToGitlab.fieldsTranslation in the code above. I named some label scopes differently in Jira and I wanted to translate them on the fly.

This also served as a whitelist of which fields I cared about inside each row.

The fieldsTranslation object was a simple key-value pair like:

{
    "Priority": "Priority",
    "Issue Type": "Type"
}

I did something similar for the values themselves. See valueToLabel in migrate.js and how it uses the fields defined in config.jiraToGitlab to work out how to translate them.

Step 2.4 Assigning issues to milestones

I don't want to lose when we did issues, so I'll create the old sprints as milestones on Gitlab.

For that I'm fetching a dashboard chart that only has the sprints and using the resulting list to create all milestones in GitLab:

// const gitlabUrl = ...
const sprintData = dashboard.getChartAsJson(654321);
const sprints = sprintData.data.map( row => row[0] ); // The chart only has one column
sprints.forEach(async title => {
    const due_date = new Date(title.split(" ")[0]); // Our sprints have the due date in their name
    const headers = {
        "Private-Token": privateToken,
        "Content-Type": "application/json"
    }
    await request.POST(
        `${gitlabUrl}/api/v4/groups/2/milestones`,
        headers,
        { title, due_date }
    );
});

Next I just need to process each issue and add the correct milestone to it:

// ...
// const jsonRows = ...
// const gitlabUrl = ...
// ...
const headers = {
    "Private-Token": privateToken,
    "Content-Type": "application/json"
}
const milestones = await request.GET(`${gitlabUrl}/api/v4/groups/2/milestones?per_page=9999999`, headers);
jsonRows.forEach(async (row) => {
    const issueCode = `${row["Project Key"]}-${row["Issue Number"]}`;
    const issues = await request.GET(`${gitlabUrl}/api/v4/groups/2/issues?search="[${issueCode}] "`, { "Private-Token": privateToken });
    const match = issues[0];
    await request.PUT(
        `${gitlabUrl}/api/v4/projects/${match["project_id"]}/issues/${match["iid"]}`, 
        headers,
        { "milestone_id":  milestones.find(m => m.title === match["Sprint"]).id }
    );
});

Wrap up

Now that GitLab contains all our issues, the next steps are:

  • Produce some dashboards over the data. For that, some data wrangling is in order.
  • Make GitLab issues visible from Zendesk.

More on both of these in the future!

 

Note: the code in this blogpost is provided as an example of we used in GitLab's, Jira's and the Pi Dashboard's APIs to assist us in the migration of the Jira issues to GitLab issues. They should serve mostly as inspiration.

The same goes for the code in the repository: it is provided as is.

You'll find some bits in the repository to assign issues to people. This is because sometimes GitLab's migration tool left issues unassigned ¯\_(ツ)_/¯.

All of that said, if you need to do something similar or just have some questions, feel free to drop me a line on twitter.

And if you want to join our team and work on interesting engineering problems - browse our job openings.

"We slowly came to the realisation that perhaps Jira was overkill for what we wanted to do and started looking into how we could use GitLab to do the same thing.

Houston... we've got mail.

Sign up with your email to receive news, updates and the latest blog articles to inspire you and your business.
  • This field is for validation purposes and should be left unchanged.
Privacy PolicyT&Cs
© Panintelligence 2021