This site uses cookies to deliver our services. By using this site, you acknowledge that you have read and understand our Cookie and Privacy policy. Your use of Kontext website is subject to this policy. Allow Cookies and Dismiss

Invoke Hadoop WebHDFS APIs in .NET Core

187 views 0 comments last modified about 5 months ago Raymond Tang

lite-log hadoop hdfs

Background

Apache doesn't provide native official .NET APIs for Hadoop HDFS. The HTTP REST API supports the complete FileSystem/FileContext interface for HDFS.

Thus, we could use these web APIs to perform HDFS operations in other programming language like C#.

WebHDFS APIs reference

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html

Examples

List files

The following code snippet retrieve the file list in the root directory in my local Hadoop node.:

static void Main(string[] args)
         {
             WebHdfsListStatusApi();
             Console.ReadLine();
         }

        static void WebHdfsListStatusApi()
         {

            var protocal = "http";
             var host = "127.0.0.1";
             var port = 9870;
             var hdfsFilePath = "\\";
             var operation = "LISTSTATUS";
             var url = $"{protocal}://{host}:{port}/webhdfs/v1/{hdfsFilePath}?op={operation}";
             var request = (HttpWebRequest)WebRequest.Create(url);
             var response = (HttpWebResponse)request.GetResponse();
             using (StreamReader reader = new StreamReader(response.GetResponseStream()))
             {
                 var result = reader.ReadToEnd();
                 Console.WriteLine(result);
             }
         }

The output looks like the following screenshot:

image

The following is the output in Postman:

image

Get file content

Similarly you can also get the content of a file through OPEN operation:

static void Main(string[] args)
         {
             WebHdfsGetFileContent();
             Console.ReadLine();
         }

        static void WebHdfsGetFileContent()
         {

            var protocal = "http";
             var host = "127.0.0.1";
             var port = 9870;
             var hdfsFilePath = "\\Sales.csv";
             var operation = "OPEN";
             var url = $"{protocal}://{host}:{port}/webhdfs/v1/{hdfsFilePath}?op={operation}";
             var request = (HttpWebRequest)WebRequest.Create(url);
             var response = (HttpWebResponse)request.GetResponse();
             using (StreamReader reader = new StreamReader(response.GetResponseStream()))
             {
                 var result = reader.ReadToEnd();
                 Console.WriteLine(result);
             }
         }

The following screenshot is the sample output:

image

Related pages

Set AttachDbFilename as Relative Path in .NET Core

18 views   0 comments last modified about 9 days ago

.NET Framework, you can use |DataDirectory| to configure connection string when connecting to SQL Server database file via attach mode: AttachDbFilename=|DataDirectory|\dbname.mdf In .NET Core, you cannot directly set SQL Server Express connec...

View detail

Set AttachDbFilename as Relative Path in .NET Core

18 views   0 comments last modified about 9 days ago

.NET Framework, you can use |DataDirectory| to configure connection string when connecting to SQL Server database file via attach mode: AttachDbFilename=|DataDirectory|\dbname.mdf In .NET Core, you cannot directly set SQL Server Express connec...

View detail

Instantiate a Service in ConfigureServices Method in .NET Core

14 views   0 comments last modified about 9 days ago

.NET Core is built in with dependency injection. Usually method ConfigureServices in Startup class is used to register services in the container. The signature of the method looks like the following: public void ConfigureServices(IServiceC...

View detail

Instantiate a Service in ConfigureServices Method in .NET Core

14 views   0 comments last modified about 9 days ago

.NET Core is built in with dependency injection. Usually method ConfigureServices in Startup class is used to register services in the container. The signature of the method looks like the following: public void ConfigureServices(IServiceC...

View detail

ASP.NET Core 2.1 Error - 'Cyrillic' is not a supported encoding name

30 views   0 comments last modified about 15 days ago

After upgrading to ASP.NET Core 2.1 (.NET Core SDK 2.1.301), you may encounter the following error about encoding: System.ArgumentException    HResult=0x80070057    Message='Cyrillic' is not a supported encoding name. For information on defining a custo...

View detail

SQLite in .NET Core with Entity Framework Core

21 views   0 comments last modified about 18 days ago

SQLite is a self-contained and embedded SQL database engine. In .NET Core, Entity Framework Core provides APIs to work with SQLite. This page provides sample code to create a SQLite database using package Microsoft.EntityFrameworkCore.Sqlite . Create sample project ...

View detail

Add comment

Please login first to add comments.  Log in New user?  Register

Comments (0)

No comments yet.