Troubleshooting Data Uploads

If you have trouble with uploading files to the Pennsieve platform, please follow the following steps to try to mitigate the issues. I these do not work, please contact the Pennsieve team through the platform support functionality. We will reach out to you to work with you to address the issues.

These steps apply to datasets that are uploaded programmatically and not to files uploaded through the web-application. We do not recommend uploading datasets through the web-application.

Initial steps to resolve issues with uploading data to Pennsieve

We recommend to try the following steps to try to address issues with the Pennsieve Agent Uploader.

📘

Be patient

If files seem to upload correctly, it is expected that they do not immediately show up in the dataset when you refresh the file-browser. It can take up to 1-2 minutes for uploaded files to be available in the dataset due to the nature of the processing pipeline.

If files do not show up after that period of time, it is possible that something is not functioning correctly. You can check the status of uploaded files for a manifest by:

1. Navigating to the activity panel in the dataset on the Pennsieve Platform
2. Selecting the Upload Manifests tab and clicking on the arrow next to the manifest you are uploading, and
3. Checking the status of the files. You can find a list expected status indicators here: Viewing dataset upload manifests

📘

Restart the Pennsieve Agent

If files do not seem to upload correctly, the first step would be to test if restarting the agent resolves the issue.

1. Call pennsieve agent stop from the command line and verify that the agent is stopped.
2. Call pennsieve agent to start the agent again.
3. Try to upload the dataset again.

📘

Verify that you have the latest version of the agent

The Pennsieve platform is in constant development and it is possible that you are using an outdated client. Please verify that you are using the latest software.

1. Check the version of the agent by calling pennsieve version
2. The agent should be running while you retrieve the version and the Agent- and Client version numbers should be the same.
3. You can find the latest version of the Agent here: https://github.com/Pennsieve/pennsieve-agent/releases.
4. If your local Pennsieve Agent version does not match the latest available version:
   a. Stop the currently running agent by calling pennsieve agent stop
   b. Download the newer Pennsieve Agent installer and install the new version.
   c. Start the agent by calling pennsieve agent
   d. Verify that the version of the agent and the CLI is updated to match the latest version.

📘

Check that you are logged in with the correct credentials and have selected the correct dataset

Pennsieve requires authentication and authorization of users as part of data upload. Ensure that you are logged in with the correct credentials and are uploading to a dataset that grants you write access.

1. Call pennsieve whoami to see which authentication profile you are using. Verify that this is associated with the user that you want to use as the uploader,
2. Call pennsieve dataset to see which dataset is currently set as the active dataset. This should be a dataset that you can access and have "write" permissions for. Update the active dataset as needed using the pennsieve dataset use <dataset_id> command.

If the whoami method fails, make sure that you have credentials setup correctly for your Pennsieve account by following: Configuring client credentials

🚧

Check the Agent Log for any clues to why the agent is not functioning as expected.

If none of the above resolves your issue, you can check the Pennsieve Agent log files for clues about what is happening. This could be anything from not having internet access, to universities blocking access to our site to unknown issues with our platform.

• You can check the log of the Agent for any additional clues to why it is not functioning as expected. The Pennsieve Agent log can be found in the ~/.pennsieve/agent.log file.

❗️

Remove the local Pennsieve Agent Database files and restart the agent.

The Pennsieve Agent stores the status of an upload manifest in a local database on the users' computer as well as the active profile and dataset. Removing these files does not affect any data on your computer or on the Pennsieve platform but does remove a local history of uploaded files. When you restart the agent, a new database will be initiated and used for subsequent uploads.

NOTE: This will locally remove any stored progress on partially uploaded datasets. That is, all manifests will be removed locally and you will need to create a new upload manifest to upload files to the platform.

1. Stop the agent by calling pennsieve agent stop
2. Go to the ~/.pennsieve folder
3. (optional) Backup the pennsieve_agent.db pennsieve_agent.db-shm pennsieve_agent.db-wal files by renaming them, or:
4. Remove the database files by calling rm rm pennsieve_agent.db pennsieve_agent.db-shm pennsieve_agent.db-wal
5. Restart the agent by calling pennsieve agent
6. Ensure you set your profile, active dataset and create/upload a new manifest

If you are not comfortable with this action, feel free to reach out to the Pennsieve team for additional assistance.

Contacting Pennsieve about technical issues regarding uploading files

If none of these steps resolve the issue you are seeing, please contact the Pennsieve team. We will need the following information to help us address the issues you are seeing

1. Please provide a short description of what you are trying to do.
2. Please specify the dataset ID in the message (N:Dataset:xxx-xxxx-x-xx)
3. Please specify the affected upload manifest in the message (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx)
4. Please specify the version of the agent that was used to upload the dataset, or version of any external tools you are using to interface with the Pennsieve Platform.
5. If possible, include screenshots of what you see and explain what you expected to see.
6. Please forward the agent.log file which can be found in the ~/.pennsieve folder to [email protected]

You can contact the Pennsieve team through the web-application, or send an email to [email protected]