星空网 > 软件开发 > 数据库

DW(六):polybase访问Azure Blob Storage

 

目录:

  • 连接hadoop配置语法
  • 配置hadoop连接
  • Pushdown配置
  • Create external tables for Azure blob storage

连接hadoop配置语法:  


global configuration settings for PolyBase Hadoop and Azure blob storage connectivity, Syntax:

  • sp_configure:  List all of the configuration options
  • 配置与hadoop连接语法:
  • sp_configure [ @configname = ] 'hadoop connectivity',        [ @configvalue = ] { 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 } [;]  RECONFIGURE [;]  

    Option 0: Disable Hadoop connectivity Option 1: Hortonworks HDP 1.3 on Windows Server Option 1: Azure blob storage (WASB[S]) Option 2: Hortonworks HDP 1.3 on Linux Option 3: Cloudera CDH 4.3 on Linux Option 4: Hortonworks HDP 2.0 on Windows Server Option 4: Azure blob storage (WASB[S]) Option 5: Hortonworks HDP 2.0 on Linux Option 6: Cloudera 5.1, 5.2, 5.3, 5.4, and 5.5 on Linux Option 7: Hortonworks 2.1, 2.2, and 2.3 on Linux Option 7: Hortonworks 2.1, 2.2, and 2.3 on Windows Server Option 7: Azure blob storage (WASB[S])

  • 示例代码:
    1. 显示所有可用的配置列表: EXEC sp_configure;
    2. 显示某个配置项的配置信息:  EXEC sp_configure @configname='hadoop connectivity';
    3. 设置配置项示例语法如下:       
    4. sp_configure @configname = 'hadoop connectivity', @configvalue = 7; GO  RECONFIGURE GO  

       

 配置hadoop连接


  • 在本节示例中,连接HDInsight3.2集,对应关系如下:
  • DW(六):polybase访问Azure Blob Storage
  • 配置polybase连接配置项信息信息如下:
  • sp_configure @configname = 'hadoop connectivity', @configvalue = 7; GO  RECONFIGURE GO

     

  • 命令行:  services.msc, 重启下面的服务
    1. SQL Server PolyBase Data Movement Service
    2. SQL Server PolyBase Engine

 Pushdown配置:


  • To improve query performance, enable pushdown computation to a Hadoop cluster
  1. Find the file yarn-site. in the installation path of SQL Server. Typically, the path is: C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Binn\Polybase\Hadoop\conf
  2. On the Hadoop machine, find the analogous(类似) file in the Hadoop configuration directory. In the file, find and copy the value of the configuration key yarn.application.classpath
  3. On the SQL Server machine, in the yarn-site. file, find the yarn.application.classpath property. Paste the value from the Hadoop machine into the value element.
  4. $HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*

     

 Create external tables for Azure blob storage


  • The Elastic Database query feature relies on the these four DDL statements. Typically, these DDL statements are used once or rarely when the schema of your application changes
    1. [CREATE MASTER KEY](https://msdn.microsoft.com/library/ms174382.aspx)
    2. [CREATE CREDENTIAL](https://msdn.microsoft.com/library/ms189522.aspx)
    3. [CREATE DATABASE SCOPED CREDENTIAL](https://msdn.microsoft.com/library/mt270260.aspx)
    4. [CREATE/DROP EXTERNAL DATA SOURCE](https://msdn.microsoft.com/library/dn935022.aspx)
    5. [CREATE/DROP EXTERNAL TABLE](https://msdn.microsoft.com/library/dn935021.aspx)
  • You can use the following syntax to drop the master key and credentials
    1. DROP CREDENTIAL <credential_name> ON DATABASE;
    2. DROP DATABASE SCOPED CREDENTIAL <credential_name>;
    3. DROP MASTER KEY; 

 




原标题:DW(六):polybase访问Azure Blob Storage

关键词:azure

*特别声明:以上内容来自于网络收集,著作权属原作者所有,如有侵权,请联系我们: admin#shaoqun.com (#换成@)。

非洲十大赛道调研报告之四:电商:https://www.ikjzd.com/articles/139757
最新国检结果出炉!玩具、童车共30批次不合格~:https://www.ikjzd.com/articles/139758
海关总署:2020年通过海关跨境电商验放进出口清单同比增长超63%:https://www.ikjzd.com/articles/139759
推特上一片静悄悄,特朗普突然去干了一件大事 !:https://www.ikjzd.com/articles/13976
Facebook | 8个技巧,让你的FB线上营销发挥最大效益:https://www.ikjzd.com/articles/139760
什么是亚马逊“飞轮理论”:https://www.ikjzd.com/articles/139761
37号文今后是否会更新?一文详解关键信息 :https://www.kjdsnews.com/a/1836441.html
探讨内地人开设香港账户的可行性 :https://www.kjdsnews.com/a/1836442.html
相关文章
我的浏览记录
海外公司注册 | 跨境电商服务平台 | 深圳旅行社 | 东南亚物流