詳解ASP.NET Core實現的文件斷點上傳下載
場景分析
文件的斷點續傳功能可以應用于需要上傳或下載大文件的場景。在網絡狀況不佳或者文件較大時,一次性上傳或下載整個文件可能會耗費大量時間和帶寬,并且可能會導致因中斷而失敗的情況發生。通過實現文件的斷點續傳功能,可以將大文件分割成小塊,分別上傳或下載,即使在網絡出現問題時也可以通過上傳或下載已經完成的文件塊來繼續未完成的操作,減少耗時和帶寬消耗,提高用戶體驗。常見的應用場景如:視頻、音頻文件的上傳或下載,大型軟件的更新或下載,云盤等文件存儲服務等等。
設計思路
在asp.net core 中實現文件的斷點續傳功能,需要進行以下步驟:
- 客戶端與服務端建立連接。
- 客戶端上傳文件時,將文件切割成小塊并分別傳輸到服務端。
- 服務端接收到文件塊后進行校驗和整合。
- 客戶端下載文件時,請求服務端指定位置開始的文件塊,并按照順序拼接成完整文件。
具體實現思路如下:
斷點上傳
- 客戶端上傳文件時,服務端接收到請求,在服務端創建對應的文件,并記錄上傳時間、文件大小等信息。
- 將接收到的文件塊保存到物理文件中,每次上傳塊的時候都應該壓縮這個塊,以減少帶寬的消耗。
- 服務端返回給客戶端成功接收的文件塊編號和文件處理情況。
斷點下載
- 客戶端下載文件時,服務端接收到請求,讀取文件對應的塊信息,判斷客戶端需要下載的文件塊范圍是否存在于服務端的文件塊信息中。
- 如果該文件塊不存在,則返回錯誤信息。
- 如果該文件塊存在,則讀取對應文件塊并返回給客戶端。
實現流程
斷點上傳
創建上傳控制器
在 ASP.NET Core 應用程序中,創建一個名為 UploadController 的控制器。在該控制器中,使用 ApiControllerAttribute 特性聲明該控制器為 Web API 控制器。
[ApiController]
[Route("[controller]")]
public class UploadController : ControllerBase
實現上傳方法
在控制器中實現下面的代碼。
[HttpPost]
public async Task<IActionResult> Upload(IFormFile file, int chunkIndex, int totalChunks)
{
if (file == null || file.Length == 0)
{
return BadRequest("This request does not have any body");
}
// create the folder if it doesn't exist yet
string folderPath = Path.Combine(Directory.GetCurrentDirectory(), "Uploads");
if (!Directory.Exists(folderPath))
{
Directory.CreateDirectory(folderPath);
}
string filePath = Path.Combine(folderPath, file.FileName);
using (FileStream stream = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.Write))
{
await file.CopyToAsync(stream);
}
return Ok();
}
實現斷點上傳邏輯
在上面的代碼中,我們只是將整個文件保存到了服務器上。現在,我們需要實現斷點上傳的邏輯。斷點上傳指的是,將大文件分成多個小塊,并逐個上傳。
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using System;
using System.IO;
using System.Threading.Tasks;
namespace ResumeTransfer
{
[Route("api/[controller]")]
[ApiController]
public class UploadController : ControllerBase
{
private const string UploadsFolder = "Uploads";
[HttpPost]
public async Task<IActionResult> Upload(IFormFile file, int? chunkIndex, int? totalChunks)
{
if (file == null || file.Length == 0)
{
return BadRequest("This request does not have any body");
}
// Check if the chunk index and total chunks are provided.
if (!chunkIndex.HasValue || chunkIndex.Value < 0 || !totalChunks.HasValue || totalChunks.Value <= 0)
{
return BadRequest("Invalid chunk index or total chunks");
}
// Create folder for upload files if not exists
string folderPath = Path.Combine(Directory.GetCurrentDirectory(), UploadsFolder);
if (!Directory.Exists(folderPath))
{
Directory.CreateDirectory(folderPath);
}
string filePath = Path.Combine(folderPath, file.FileName);
// Check if chunk already exists
if (System.IO.File.Exists(GetChunkFileName(filePath, chunkIndex.Value)))
{
return Ok();
}
using (FileStream stream = new FileStream(GetChunkFileName(filePath, chunkIndex.Value), FileMode.Create, FileAccess.Write, FileShare.Write))
{
await file.CopyToAsync(stream);
}
if (chunkIndex == totalChunks - 1)
{
// All the chunks have been uploaded, merge them into a single file
MergeChunks(filePath, totalChunks.Value);
}
return Ok();
}
private void MergeChunks(string filePath, int totalChunks)
{
using (var finalStream = new FileStream(filePath, FileMode.CreateNew))
{
for (int i = 0; i < totalChunks; i++)
{
var chunkFileName = GetChunkFileName(filePath, i);
using (var stream = new FileStream(chunkFileName, FileMode.Open))
{
stream.CopyTo(finalStream);
}
System.IO.File.Delete(chunkFileName);
}
}
}
private string GetChunkFileName(string filePath, int chunkIndex)
{
return $"{filePath}.part{chunkIndex.ToString().PadLeft(5, '0')}";
}
}
}
客戶端上傳
客戶端使用 axios 庫進行文件上傳。使用下面的代碼,將文件分塊并管理每個文件塊的大小和數量。
// get file size and name
const fileSize = file.size;
const fileName = file.name;
// calculate chunk size
const chunkSize = 10 * 1024 * 1024; // 10MB
// calculate total chunks
const totalChunks = Math.ceil(fileSize / chunkSize);
// chunk upload function
const uploadChunk = async (chunkIndex) => {
const start = chunkIndex * chunkSize;
const end = Math.min((chunkIndex + 1) * chunkSize, fileSize);
const formData = new FormData();
formData.append("file", file.slice(start, end));
formData.append("chunkIndex", chunkIndex);
formData.append("totalChunks", totalChunks);
await axios.post("/upload", formData);
};
for (let i = 0; i < totalChunks; i++) {
await uploadChunk(i);
}
斷點下載
創建下載控制器
創建一個名為 DownloadController 的控制器,使用 ApiControllerAttribute 特性聲明該控制器為 Web API 控制器。
[ApiController]
[Route("[controller]")]
public class DownloadController : ControllerBase
實現斷點下載邏輯
在控制器中實現下面的代碼。
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using System;
using System.IO;
namespace ResumeTransfer
{
[Route("api/[controller]")]
[ApiController]
public class DownloadController : ControllerBase
{
private const string UploadsFolder = "Uploads";
[HttpGet("{fileName}")]
public IActionResult Download(string fileName, long? startByte = null, long? endByte = null)
{
if (string.IsNullOrEmpty(fileName))
{
return BadRequest("Invalid file name");
}
string filePath = Path.Combine(Directory.GetCurrentDirectory(), UploadsFolder, fileName);
if (!System.IO.File.Exists(filePath))
{
return NotFound();
}
long contentLength = new System.IO.FileInfo(filePath).Length;
// Calculate the range to download.
if (startByte == null)
{
startByte = 0;
}
if (endByte == null)
{
endByte = contentLength - 1;
}
// Adjust the startByte and endByte to be within the range of the file size.
if (startByte.Value < 0 || startByte.Value >= contentLength || endByte.Value < startByte.Value || endByte.Value >= contentLength)
{
Response.Headers.Add("Content-Range", $"bytes */{contentLength}");
return new StatusCodeResult(416); // Requested range not satisfiable
}
// Set the Content-Disposition header to enable users to save the file.
Response.Headers.Add("Content-Disposition", $"inline; filename={fileName}");
Response.StatusCode = 206; //Partial Content
Response.Headers.Add("Accept-Ranges", "bytes");
long length = endByte.Value - startByte.Value + 1;
Response.Headers.Add("Content-Length", length.ToString());
// Send the file data in a range of bytes, if requested
byte[] buffer = new byte[1024 * 1024];
using (FileStream stream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
stream.Seek(startByte.Value, SeekOrigin.Begin);
int bytesRead;
while (length > 0 && (bytesRead = stream.Read(buffer, 0, (int)Math.Min(buffer.Length, length))) > 0)
{
// Check if the client has disconnected.
if (!Response.HttpContext.Response.Body.CanWrite)
{
return Ok();
}
Response.Body.WriteAsync(buffer, 0, bytesRead);
length -= bytesRead;
}
}
return new EmptyResult();
}
}
}
客戶端下載
客戶端使用 axios 庫進行文件下載。使用下面的代碼,將要下載的文件拆分成小塊,并按照順序下載。
const CHUNK_SIZE = 1024 * 1024 * 5; // 5MB
const downloadChunk = async (chunkIndex) => {
const res = await axios.get(`/download?fileName=${fileName}&chunkIndex=${chunkIndex}`, {
responseType: "arraybuffer",
});
const arrayBuffer = res.data;
const start = chunkIndex * CHUNK_SIZE;
const end = start + CHUNK_SIZE;
const blob = new Blob([arrayBuffer]);
const url = URL.createObjectURL(blob);
const a = document.createElement("a");
a.style.display = "none";
a.href = url;
a.download = fileName;
document.body.appendChild(a);
a.click();
URL.revokeObjectURL(url);
};
for (let i = 0; i < totalChunks; i++) {
await downloadChunk(i);
}
單元測試
在我們的代碼中實現單元測試可以確保代碼的正確性,并且可以減少手工測試的負擔。我們可以使用
Microsoft.VisualStudio.TestTools.UnitTesting(在 .NET Core 中,也可以使用 xUnit 或 NUnit 進行單元測試)進行單元測試。
下面是一個簡單的上傳控制器單元測試示例:
using ResumeTransfer;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using System.IO;
using System.Threading.Tasks;
using Xunit;
using Microsoft.Extensions.Primitives;
using Moq;
namespace ResumeTransfer.Tests
{
public class UploadControllerTests
{
[Fact]
public async Task Upload_ReturnsBadRequest_WhenNoFileIsSelected()
{
// Arrange
var formCollection = new FormCollection(new Dictionary<string, StringValues>(), new FormFileCollection());
var context = new Mock<HttpContext>();
context.SetupGet(x => x.Request.Form).Returns(formCollection);
var controller = new UploadController { ControllerContext = new ControllerContext { HttpContext = context.Object } };
// Act
var result = await controller.Upload(null, 0, 1);
// Assert
Assert.IsType<BadRequestObjectResult>(result);
}
[Fact]
public async Task Upload_ReturnsBadRequest_WhenInvalidChunkIndexOrTotalChunksIsProvided()
{
// Arrange
var fileName = "test.txt";
var fileStream = new MemoryStream(System.Text.Encoding.UTF8.GetBytes("This is a test file."));
var formFile = new FormFile(fileStream, 0, fileStream.Length, "Data", fileName);
var formCollection = new FormCollection(new Dictionary<string, StringValues>
{
{ "chunkIndex", "0" },
{ "totalChunks", "0" }
}, new FormFileCollection { formFile });
var context = new Mock<HttpContext>();
context.SetupGet(x => x.Request.Form).Returns(formCollection);
var controller = new UploadController { ControllerContext = new ControllerContext { HttpContext = context.Object } };
// Act
var result = await controller.Upload(formFile, 0, 0);
// Assert
Assert.IsType<BadRequestObjectResult>(result);
}
[Fact]
public async Task Upload_UploadsChunk_WhenChunkDoesNotExist()
{
// Arrange
var fileName = "test.txt";
var fileStream = new MemoryStream(System.Text.Encoding.UTF8.GetBytes("This is a test file."));
var formFile = new FormFile(fileStream, 0, fileStream.Length, "Data", fileName);
var formCollection = new FormCollection(new Dictionary<string, StringValues>
{
{ "chunkIndex", "0" },
{ "totalChunks", "1" }
}, new FormFileCollection { formFile });
var context = new Mock<HttpContext>();
context.SetupGet(x => x.Request.Form).Returns(formCollection);
var uploadsFolder = Path.Combine(Directory.GetCurrentDirectory(), "uploads");
var controller = new UploadController { ControllerContext = new ControllerContext { HttpContext = context.Object }, UploadsFolder = uploadsFolder };
// Act
var result = await controller.Upload(formFile, 0, 1);
var uploadedFilePath = Path.Combine(uploadsFolder, fileName);
// Assert
Assert.IsType<OkResult>(result);
Assert.True(File.Exists(uploadedFilePath));
using (var streamReader = new StreamReader(File.OpenRead(uploadedFilePath)))
{
var content = await streamReader.ReadToEndAsync();
Assert.Equal("This is a test file.", content);
}
File.Delete(uploadedFilePath);
}
[Fact]
public async Task Upload_UploadsChunk_WhenChunkExists()
{
// Arrange
var fileName = "test.txt";
var fileStream = new MemoryStream(System.Text.Encoding.UTF8.GetBytes("This is a test file."));
var formFile = new FormFile(fileStream, 0, fileStream.Length, "Data", fileName);
var formCollection1 = new FormCollection(new Dictionary<string, StringValues>
{
{ "chunkIndex", "0" },
{ "totalChunks", "2" }
}, new FormFileCollection { formFile });
var context = new Mock<HttpContext>();
context.SetupGet(x => x.Request.Form).Returns(formCollection1);
var uploadsFolder = Path.Combine(Directory.GetCurrentDirectory(), "uploads");
var controller = new UploadController { ControllerContext = new ControllerContext { HttpContext = context.Object }, UploadsFolder = uploadsFolder };
// Act
var result1 = await controller.Upload(formFile, 0, 2);
// Assert
Assert.IsType<OkResult>(result1);
// Arrange
var formCollection2 = new FormCollection(new Dictionary<string, StringValues>
{
{ "chunkIndex", "1" },
{ "totalChunks", "2" }
}, new FormFileCollection { formFile });
context.SetupGet(x => x.Request.Form).Returns(formCollection2);
// Act
var result2 = await controller.Upload(formFile, 1, 2);
var uploadedFilePath = Path.Combine(uploadsFolder, fileName);
// Assert
Assert.IsType<OkResult>(result2);
Assert.True(File.Exists(uploadedFilePath));
using (var streamReader = new StreamReader(File.OpenRead(uploadedFilePath)))
{
var content = await streamReader.ReadToEndAsync();
Assert.Equal("This is a test file.This is a test file.", content);
}
File.Delete(uploadedFilePath);
}
}
}
性能測試
為了實現更好的性能和響應時間,我們可以使用 BenchmarkDotNet 進行性能測試,以便找到性能瓶頸并對代碼進行優化。
下面是一個簡單的上傳控制器性能測試示例:
using BenchmarkDotNet.Attributes;
using ResumeTransfer;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http.Internal;
namespace ResumeTransfer.Benchmarks
{
[MemoryDiagnoser]
public class UploadControllerBenchmarks
{
private readonly UploadController _controller;
private readonly IFormFile _testFile;
public UploadControllerBenchmarks()
{
_controller = new UploadController();
_testFile = new FormFile(new MemoryStream(System.Text.Encoding.UTF8.GetBytes("This is a test file")), 0, 0, "TestFile", "test.txt");
}
[Benchmark]
public async Task<IActionResult> UploadSingleChunk()
{
var formCollection = new FormCollection(new System.Collections.Generic.Dictionary<string, Microsoft.Extensions.Primitives.StringValues>
{
{ "chunkIndex", "0" },
{ "totalChunks", "1" }
}, new FormFileCollection { _testFile });
var request = new DefaultHttpContext();
request.Request.Form = formCollection;
_controller.ControllerContext = new ControllerContext { HttpContext = request };
return await _controller.Upload(_testFile, 0, 1);
}
[Benchmark]
public async Task<IActionResult> UploadMultipleChunks()
{
var chunkSizeBytes = 10485760; // 10 MB
var totalFileSizeBytes = 52428800; // 50 MB
var totalChunks = (int)Math.Ceiling((double)totalFileSizeBytes / chunkSizeBytes);
for (var i = 0; i < totalChunks; i++)
{
var chunkStartByte = i * chunkSizeBytes;
var chunkEndByte = Math.Min(chunkStartByte + chunkSizeBytes - 1, totalFileSizeBytes - 1);
var chunkFileContent = new byte[chunkEndByte - chunkStartByte + 1];
using (var memoryStream = new MemoryStream(chunkFileContent))
{
using (var binaryWriter = new BinaryWriter(memoryStream))
{
binaryWriter.Write(chunkStartByte);
binaryWriter.Write(chunkEndByte);
}
}
var chunkFileName = $"{_testFile.FileName}.part{i.ToString().PadLeft(5, '0')}";
var chunkFilePath = Path.Combine(Directory.GetCurrentDirectory(), ChunkUploadController.UploadsFolder, chunkFileName);
using (var fileStream = new FileStream(chunkFilePath, FileMode.Create, FileAccess.Write, FileShare.Write))
{
await fileStream.WriteAsync(chunkFileContent);
await fileStream.FlushAsync();
}
}
var formCollection = new FormCollection(new System.Collections.Generic.Dictionary<string, Microsoft.Extensions.Primitives.StringValues>
{
{ "chunkIndex", "0" },
{ "totalChunks", totalChunks.ToString() }
}, new FormFileCollection { _testFile });
var request = new DefaultHttpContext();
request.Request.Form = formCollection;
_controller.ControllerContext = new ControllerContext { HttpContext = request };
return await _controller.Upload(_testFile, 0, totalChunks);
}
}
}
運行上面的代碼后,將會輸出詳細的性能測試結果。
根據測試結果,我們可以找到性能瓶頸,并對代碼進行優化,以達到更高的性能和更快的響應時間。