Linux CentOS 大数据集群的基础配置(2) 手动编译特定版本CHD

开始之前确保你具备以下条件：1，使用VMware中的一台虚拟机。2，虚拟机操作系统为CentOS 6.9 64位 basic server版本。3，4G及以上内存，40G及以上硬盘。4，虚拟机可以正常的访问互联网。关闭防火墙serviceiptablesstopchkconfigiptablesoff关闭selinuxvim /etc/selinux/c...

大数据流浪法师

482人浏览 · 2019-10-31 20:47:28

大数据流浪法师 · 2019-10-31 20:47:28 发布

开始之前确保你具备以下条件(劝退警告)：
1，使用VMware中的一台虚拟机。
2，虚拟机操作系统为CentOS 6.9 64位 basic server版本。
3，每台虚拟机4G及以上内存，40G及以上硬盘。
4，虚拟机可以正常的访问互联网。

此安装记录只适合上述版本，其它版本不兼容，仅供参考。
严格地执行此安装记录，将会消耗你1小时左右的时间。

关闭防火墙

service  iptables   stop
chkconfig   iptables  off

在这里插入图片描述

关闭selinux

vim /etc/selinux/config

把SELINUX=enforcing注释，添加SELINUX=disabled
在这里插入图片描述

卸载已安装的jdk

首先查询已安装的jdk

rpm -qa | grep java

在这里插入图片描述
把查询出来的这些全部卸载掉

rpm -e tzdata-java-2016j-1.el6.noarch java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el6_8.x86_64 java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el6_8.x86_64

在这里插入图片描述
再次查询，看看是否完全卸载了

rpm -qa | grep java

在这里插入图片描述
ok，全部卸载完成

安装jdk1.7.0_71

首先创建两个目录

mkdir -p /export/servers
mkdir -p /export/softwares

在这里插入图片描述
使用rz将jdk安装包上传到/export/softwares
jdk1.7.0_71下载地址：
https://www.oracle.com/technetwork/java/javase/archive-139210.html
如果没有rz命令，可以输入以下命令安装lrzsz，然后就能用rz命令了。

yum -y install lrzsz

在这里插入图片描述
安装lrzsz成功，现在切换到softwares目录，开始上传

cd /export/softwares/
rz

在这里插入图片描述

上传完成，把它解压到/export/servers/目录下

cd /export/softwares/
tar -zxvf jdk-7u71-linux-i586.tar.gz -C /export/servers/

配置环境变量：

vim /etc/profile.d/java.sh

输入以下内容保存退出

export JAVA_HOME=/export/servers/jdk1.7.0_71
export PATH=$PATH:$JAVA_HOME/bin

source一下profile，重新加载环境变量

source /etc/profile

输入如下命令，检查jdk1.7.0_71是否安装成功

java -version

在这里插入图片描述

安装maven3.0.5

在这里下载maven3.0.5
https://archive.apache.org/dist/maven/maven-3/

将maven的安装包上传到/export/softwares
在这里插入图片描述
然后解压maven的安装包到/export/servers

cd /export/softwares/
tar -zxvf apache-maven-3.0.5-bin.tar.gz -C ../servers/

在这里插入图片描述
然后配置maven的环境变量

vim /etc/profile.d/maven.sh

输入以下内容，保存退出

export MAVEN_HOME=/export/servers/apache-maven-3.0.5
export MAVEN_OPTS="-Xms4096m -Xmx4096m"
export PATH=$PATH:$MAVEN_HOME/bin

让修改立刻生效

配置MAVEN仓库

上传本地仓库的压缩文件到/export/softwares/

在这里插入图片描述
把它解压到/export/servers/

cd /export/softwares/
unzip repository.zip -d ../servers/

在这里插入图片描述
修改maven的配置文件

cd  /export/servers/apache-maven-3.0.5/conf
vim settings.xml

找到
在这里插入图片描述
修改为如下内容：

再查找mirrors

修改为下图内容，换成阿里云的镜像

保存退出，maven本地仓库和阿里源就配置好了。

安装findbugs1.3.9

下载地址：
https://sourceforge.net/projects/findbugs/files/findbugs/1.3.9/
下载完成后，把它上传到/export/softwares/目录下
在这里插入图片描述
然后把它解压到/export/servers/目录下

cd /export/softwares/
tar -zxvf findbugs-1.3.9.tar.gz -C ../servers/

在这里插入图片描述
配置findbugs的环境变量

vim /etc/profile.d/findbugs.sh

输入以下内容保存退出

export FINDBUGS_HOME=/export/servers/findbugs-1.3.9
export PATH=$PATH:$FINDBUGS_HOME/bin

让修改立即生效

source /etc/profile

安装一些依赖包

yum install -y autoconf automake libtool cmake
yum install -y ncurses-devel
yum install -y openssl-devel
yum install -y lzo-devel zlib-devel gcc gcc-c++

# bzip2压缩需要的依赖包
yum install -y  bzip2-devel

安装protobuf2.5.0

下载地址：
https://github.com/protocolbuffers/protobuf/releases

下载完成后上传到/export/softwares/目录下
在这里插入图片描述
上传完成后，解压到/export/servers/目录下

cd /export/softwares/
tar -zxvf protobuf-2.5.0.tar.gz -C ../servers/

在这里插入图片描述
进入它的目录里

cd   /export/servers/protobuf-2.5.0

执行如下命令

./configure

在这里插入图片描述
再执行如下命令

make && make install

在这里插入图片描述
protobuf安装完成。

安装snappy1.1.1

下载地址：
官网http://google.github.io/snappy/

Snappy1.1.1下载链接，随时可能失效

下载完后上传到/export/servers/目录下
在这里插入图片描述
上传完成后，解压到/export/servers/目录下

cd /export/softwares/
tar -zxvf snappy-1.1.1.tar.gz  -C ../servers/

在这里插入图片描述
切换到解压的目录下

cd /export/servers/snappy-1.1.1/

执行如下命令

./configure

在这里插入图片描述
执行如下命令

make && make install

在这里插入图片描述

下载cdh源码准备编译

下载地址：
http://archive.cloudera.com/cdh5/cdh/5/Hadoop-2.6.0-cdh5.14.0-src.tar.gz

下载完后上传到/export/softwares/目录下
在这里插入图片描述
上传完成后，解压到/export/servers/目录下

cd /export/softwares/
tar -zxvf hadoop-2.6.0-cdh5.14.0-src.tar.gz -C ../servers/

在这里插入图片描述
进入它的目录

cd  /export/servers/hadoop-2.6.0-cdh5.14.0

下面步骤需要联网

编译不支持snappy压缩（不执行这个）：

mvn package -Pdist,native -DskipTests –Dtar

编译支持snappy压缩(我们执行这个！)：

mvn package -DskipTests -Pdist,native -Dtar -Drequire.snappy -e -X

然而很遗憾，编译失败了，由于时间原因，这个记录被保留下来，以后有时间了，再去解决这个bug

[root@test01 hadoop-2.6.0-cdh5.14.0]# mvn package -DskipTests -Pdist,native -Dtar -D
require.snappy -e -X
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28+0800)
Maven home: /export/servers/apache-maven-3.0.5
Java version: 1.7.0_71, vendor: Oracle Corporation
Java home: /export/servers/jdk1.7.0_71/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-696.el6.x86_64", arch: "amd64", family: "unix"
[INFO] Error stacktraces are turned on.
[DEBUG] Reading global settings from /export/servers/apache-maven-3.0.5/conf/settings.xml
[DEBUG] Reading user settings from /root/.m2/settings.xml
[DEBUG] Using local repository at /root/.m2/repository
[DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10 for /root/.m2/repository
[INFO] Scanning for projects...
[DEBUG] Using connector WagonRepositoryConnector with priority 0 for https://repository.cloudera.com/artifactory/cloudera-repos
Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom
[DEBUG] Writing resolution tracking file /root/.m2/repository/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom.lastUpdated
[DEBUG] Writing resolution tracking file /root/.m2/repository/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom.lastUpdated
[DEBUG] Using connector WagonRepositoryConnector with priority 0 for http://repository.jboss.org/nexus/content/groups/public/
Downloading: http://repository.jboss.org/nexus/content/groups/public/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom
[DEBUG] Writing resolution tracking file /root/.m2/repository/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom.lastUpdated
[DEBUG] Using connector WagonRepositoryConnector with priority 0 for http://repo.maven.apache.org/maven2
Downloading: http://repo.maven.apache.org/maven2/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom
[DEBUG] Writing resolution tracking file /root/.m2/repository/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom.lastUpdated
[ERROR] The build could not read 1 project -> [Help 1]
org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.14.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): peer not authenticated and 'parent.relativePath' points at wrong local POM @ line 19, column 11

        at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:363)
        at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:636)
        at org.apache.maven.DefaultMaven.getProjectsForMavenReactor(DefaultMaven.java:585)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:234)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
[ERROR]   
[ERROR]   The project org.apache.hadoop:hadoop-main:2.6.0-cdh5.14.0 (/export/servers/hadoop-2.6.0-cdh5.14.0/pom.xml) has 1 error
[ERROR]     Non-resolvable parent POM: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.14.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): peer not authenticated and 'parent.relativePath' points at wrong local POM @ line 19, column 11 -> [Help 2]
org.apache.maven.model.resolution.UnresolvableModelException: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.14.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): peer not authenticated
        at org.apache.maven.project.ProjectModelResolver.resolveModel(ProjectModelResolver.java:159)
        at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:813)
        at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:664)
        at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:310)
        at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:232)
        at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:410)
        at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:379)
        at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:343)
        at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:636)
        at org.apache.maven.DefaultMaven.getProjectsForMavenReactor(DefaultMaven.java:585)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:234)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.sonatype.aether.resolution.ArtifactResolutionException: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.14.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): peer not authenticated
        at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:538)
        at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:216)
        at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:193)
        at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:286)
        at org.apache.maven.project.ProjectModelResolver.resolveModel(ProjectModelResolver.java:155)
        ... 22 more
Caused by: org.sonatype.aether.transfer.ArtifactTransferException: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.14.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): peer not authenticated
        at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:951)
        at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:941)
        at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:669)
        at org.sonatype.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:60)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.maven.wagon.TransferFailedException: peer not authenticated
        at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:892)
        at org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:116)
        at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88)
        at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61)
        at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:601)
        ... 4 more
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
        at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:421)
        at org.apache.maven.wagon.providers.http.httpclient.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:126)
        at org.apache.maven.wagon.providers.http.httpclient.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
        at org.apache.maven.wagon.shared.http4.ConfigurableSSLSocketFactoryDecorator.connectSocket(ConfigurableSSLSocketFactoryDecorator.java:64)
        at org.apache.maven.wagon.providers.http.httpclient.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        at org.apache.maven.wagon.providers.http.httpclient.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.maven.wagon.providers.http.httpclient.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
        at org.apache.maven.wagon.providers.http.httpclient.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
        at org.apache.maven.wagon.providers.http.httpclient.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.maven.wagon.providers.http.httpclient.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.execute(AbstractHttpClientWagon.java:746)
        at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:886)
        ... 8 more
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
[root@test01 hadoop-2.6.0-cdh5.14.0]#

编译完成之后我们需要的压缩包就在下面这个路径里面
/export/servers/hadoop-2.6.0-cdh5.14.0/hadoop-dist/target

技术共进，成长同行——讯飞AI开发者社区

更多推荐

论文笔记：AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models（AlphaEdit）

论文发表于人工智能顶会ICLR（基于定位和修改的模型编辑方法（针对和等）会破坏LLM中最初保存的知识，特别是在顺序编辑场景。为此，本文提出AlphaEdit：1、在将保留知识应用于参数之前，将扰动投影到保留知识的零空间上。2、从理论上证明，这种预测确保了在查询保留的知识时，编辑后的LLM的输出保持不变，从而减轻中断问题。3、对各种LLM（包括LLaMA3、GPT2XL和GPT-J）的广泛实验表明，