Big data interview questions
Big Data Interview Questions
JAVA related
1-1 ) What is the difference between List and Set ?The problem of getting old is still a commonplace here: List features: elements are placed in order, elements can be repeated, Set features: elements are not placed in order, elements cannot be repeated .
1-2 ) Three paradigms of databases?
Atomicity, consistency, uniqueness
1-3 ) Java's io illustration class
1-4 ) Difference between object and reference object
Object is a good object is not initialized, the referenced object even if the object has been initialized, the initialization can make your own direct new can also be directly to other assignments, then back new or back other assignment we call a reference to the object, The biggest difference is
1-5 ) Talk about your understanding of reflection mechanism and its use?
There are three ways of obtaining the reflection, namely: the forName / getClass / directly class manner using reflection may obtain an instance of the class
1-6 ) List at least five design patterns
Design methods include factory method, lazy loading, observer mode, static factory, iterator mode, appearance mode
1-7 ) RPC principle?
R pc is divided into a synchronous call and a call. The difference between asynchronous and synchronous is whether to wait for the return value from the server. R pc components include RpcServer, RpcClick, RpcProxy, RpcConnection, RpcChannel, RpcProtocol, RpcInvoker and other components.
1-8 ) What are the differences and advantages and disadvantages of ArrayList , Vector , LinkedList ? What are the differences and advantages and disadvantages of HashMap and HashTable ?
ArrayList and Vector use arrays to store data, and access the elements according to the index, both can Automatically expand the internal data length as needed to add and insert elements, both allow direct indexing of elements, but Inserting data involves memory operations such as moving array elements, so index data is fast and insert data is slow, and they are the largest.
The difference is synchronized synchronized use.
LinkedList uses a doubly linked list for storage. Indexing data by sequence number requires forward or backward traversal, but
When inserting data, you only need to record the items before and after this item, so the insertion is faster!
If you are only looking for elements at a specific position or only add or remove elements at the end of the collection, then use Vector
Or ArrayList is fine . If it is an insert or delete operation to other specified positions, it is best to choose LinkedList
The differences between HashMap and HashTable and their advantages and disadvantages:
The method in HashTable is a synchronous HashMap method, which is asynchronous by default, so an extra synchronization mechanism is required in a multi-threaded environment.
HashTable not allowed null value key and value are not allowed, but HashMap allows null value key and value are thus allowed HashMap use containKey () to determine whether there is a key.
HashTable uses Enumeration and HashMap uses iterator .
Hashtable is a subclass of Dictionary , and HashMap is an implementation of the Map interface.
1-9 ) Use StringBuffer instead of String
When you need to operate on a string, use StringBuffer instead of String . String is read-only . If you modify it, it will generate a temporary object, while StringBuffer is modifiable and will not generate a temporary object.
1-10 ) Expansion of collection
ArrayList list = new ArrayList (90000); How many times does the list expand? ?
public ArrayList () {
this (10);
}
The default extension is 10
1-11 ) Java unpacking and packetization issues
System. Out .println ( "5" + 2);
52
1-12 ) the Java in Class.forName and ClassLoader.loadClass difference
Class.forName ("xx.xx") is equivalent to Class.forName ("xx.xx", true, CALLClass.class.getClassLoader ()) . The second parameter (bool) indicates whether to initialize the class when loading the class. That is, the statement that calls the static block of the class and initializes the static member variables.
ClassLoader loader = Thread.currentThread.getContextClassLoader (); // Can also be used (ClassLoader.getSystemClassLoader ())
Class cls = loader.loadClass ("xx.xx"); // This sentence did not perform initialization forName can control whether the class is initialized, and loadClass is not initialized when loaded.
1-13 ) Difference between hashMap and hashTable
HashMap Hashtable
Parent class AbstractMap Dictiionary
Whether synchronization No is
k , v can be null yes no
The hash / rehash algorithms used by Hashtable and HashMap are about the same, so there will not be a big difference in performance.
1-14 ) How to implement array inversion
ArrayList arrayList = new ArrayList ();
arrayList.add ("A");
arrayList.add ("B");
Inverting an array
Collections.reverse (arrayList);
1-15 ) Please use Java to implement binary search
The average interviewer is always looking at your ideas, so in general, you only need to write down your ideas.
The specific implementation is as follows:
The binary search is a half search, if you want to halve, you must sort the original data to find it easily:
The implementation code is as follows:
public static int binarySearch (int [] srcArray, int des) {
int low = 0;
int high = srcArray.length-1;
while (low <= high) {
int middle = (low + high) / 2;
if (des == srcArray [middle]) {
return middle;
} else if (des <srcArray [middle]) {
high = middle-1;
} else {
low = middle + 1;
}
}
return -1;
}
1-16 ) How do two threads in Java wait for a thread to finish executing
You can use the join keyword
1-17 ) Differences in the use of hashmap hashtable currentHashMap
The difference between hashmap and hashtable is that hashtable is thread-safe, while hashmap is not thread-safe, and currentHashMap is also thread-safe.
ConcurrentHashMap uses lock segmentation technology to ensure thread safety. The segmented technology is: the data is divided into segments and stored, and a lock is added to each segment of data. When a thread accesses one data, other data can be accessed.
1-18 ) Describe briefly the gc mechanism of java , the commonly used JAVA tuning methods, how OOM is generated, and how to deal with OOM problems? ? ?
1. The program will generate a lot of object information when the program is running. When the information of these objects is not useful, it will be recycled by gc.
2. The tuning method is mainly to adjust the memory size of the young and old generations.
3, OOM is OutOfMemory abbreviated ( made with more on the tall like ) is to create a thread more, no timely recovery over the generated code is as follows:
public class JavaVMStackOOM {
private void dontStop () {
while (true) {
}
}
public void stackLeakByThread () {
while (true) {
Thread thread = new Thread (new Runnable () {
@Override
public void run () {
dontStop ();
}
});
thread.start ();
}
}
public static void main (String [] args) {
JavaVMStackOOM oom = new JavaVMStackOOM ();
oom.stackLeakByThread ();
}
4. Now that you know the above phenomenon, you should pay attention when writing the code, not to create too many threads.
Linux related
2-1 ) Turn off unnecessary servicesA. Use the ntsysv command to view the services opened and closed
B. Stop the print service
[root @ hadoop1 /] # /etc/init.d/cups stop
[root @ hadoop1 /] # chkconfig cups off
2-2 ) Turn off IP6
[root @ hadoop1 /] # vim /etc/modprobe.conf
Add the following configuration:
alias net-pf-10 off
alias ipv6 off
2-3 ) Adjust the maximum number of open files
View the current number of files: [root @ hadoop1 /] # ulimit -a
Modify the configuration:
[root @ hadoop1 /] # vi /etc/security/limits.conf Add at the end of the file:
* soft nofile 65535
* hard nofile 65535
* soft nproc 65535
* hard nproc 65535
2-4 ) Modify linux kernel parameters
[root @ hadoop1 /] # vi /etc/sysctl.conf
Append the text to the end:
net.core.somaxconn = 32768
Indicates that the swap area is used when the physical memory usage is 90% ( 100-10 = 90 )
2-5 ) Turn off noatime
Add something at the end
/ dev / sda2 / data ext3 noatime, nodiratime 0 0
2-6) Please use shell command to distribute all files under one file to other machines
Scp -r / user / local hadoop2: / user / local
2-7 ) What does echo 1 + 1 && echo "1 + 1" output
[root @ hadoop1 ~] # echo 1 + 1 && echo "1 + 1"
1 + 1
1 + 1
[root @ hadoop1 ~] # echo 1 + 1 && echo "1 + 1" && echo "1+" 1
1 + 1
1 + 1
1+ 1
2-8 ) contained grandmother to find out the amount in the current directory a and the amount of the file size is greater than 55K file
[root @ hadoop1 test] # find. | grep -ri "a"
a.text: a
I did n’t write the second sentence. I have time to do it.
2-9 ) What command does Linux use to view CPU, hard disk, and memory information?
Top command
Hadoop related
3-1 ) A brief overview of the principles of hdfs and the responsibilities of each module1. The client sends a request to nameNode to upload a file
2 , the NameNode returned to state whether the user can upload data
3. To join the client, you need to upload a 1024M file. The client will request the NameNode through Rpc , and return those DataNodes that need to be uploaded ( the distance of the allocated machine and the size of the space, etc. ). Namonode will choose the nearest principle to allocate the machine.
4 , the client requests the establishment of a block transmission pipeline chnnel upload data
5 , the upload is datanode establishes a connection with other equipment and transfers the data block to the other machines
6 , Datanode to namenode report their situation and their own information storage
7. After the first file is uploaded, perform other copy transmissions.
3-2 ) Working principle of mr
1. When the mr program is executed, a job will be executed
2.The client's jobClick will request the namenode 's jobTracker to execute the task
3.jobClick will copy the resource files of the job to the HDFS side
4, the client jobClick will to namenode submit jobs , let namenode preparation
5, Namenode of jobTracker will go to initialize the object created
6.Namenode will get the partition of hdfs
7. Namenode checks the heartbeat information of TaskTracker to see the surviving machines
8, when executed datenode perform tasks Datenode go HDFS file access to resources of the job
9. TaskTracker will execute code and log into the execution channel of JVM
10, JVM or executed MapTask or ReduceTask
11.End of execution
3-3 ) How to determine when a file exists
This is the knowledge on linux , just need to add -f parameter in IF [-f] brackets to determine whether the file exists
3-4 ) What is the difference between fsimage and edit ?
Everyone knows the relationship between the namenode and the secondary namenode . When they want to synchronize their data, they call checkpoint, and they use fsimage and edit. File to save metadata information, this new file is edit, and edit will roll back the latest data.
3-5 ) How many blocks are saved in hdfs by default?
Both hadoop1.x and hadoop2.x save three copies by default, which can be modified by the parameter dfs.replication . The number of copies should be determined according to the number of machines.
3-6 ) List several configuration file optimizations?
Core-site.xml file optimization
fs.trash.interval
Default value: 0
Note: This is the option to enable the automatic transfer of hdfs file deletion to the trash can. The value is the time to clear the trash can file. It is generally better to open this to prevent important files from being deleted by mistake. The unit is minute.
dfs.namenode.handler.count
Default: 10
Note: The number of task threads started in the hadoop system is changed to 40 here . You can also try to set the most appropriate value for the effect of this value on the efficiency change.
mapreduce.tasktracker.http.threads
Default: 40
Explanation: map and reduce are used for data transmission through http . This is to set the number of parallel threads for transmission.
3-7) Talk about data skew, how it happened, and give an optimization plan
The tilt of the data is mainly because the amount of difference between the two data is not at one level. When you only want to perform a task, it causes the tilt of the data. You can use the partitioning method to reduce the tilting performance of the reduce data, such as sampling and range partitioning. Custom partition, custom side with slanted data size
3-8 ) Briefly outline the steps to install hadoop
1. Create a hadoop account.
2.setup.Change IP.
3. Install java, modify the / etc / profile file, and configure java environment variables.
4. Modify the host file domain name.
5. Install SSH and configure keyless communication.
6. Uncompress hadoop.
7. Configure hadoop-env.sh, core-site.sh, mapre-site.sh, hdfs-site.sh under the conf file.
8. Configure hadoop environment variables.
9.Hadoop namenode -format
10.Start-all.sh
3-9 ) A brief overview of the roles and functions in hadoop
N amenode: information responsible for managing metadata
SecondName: cold backup of namenode, for the machine of namenode, it can quickly switch to the specified Secondname
DateNode: mainly used to store data.
JobTracker: manage tasks and assign tasks to taskTasker
TaskTracker: Performing tasks
3-10 ) How to quickly kill a job
1, performs hadoop job -list get job-id
2, H adoop Job the kill-ID Hadoop
3-11) How to start quickly when adding a node
Hadoop-daemon.sh start datanode
3-12) What do you think of the advantages of developing map / reduce using java, streaming, and pipe ?
I have only used java and hive to develop mapReduce, but it is awkward and slow to develop mapreduce using java. Because of the slowness of java, hive is used, which facilitates query and design
3-13 ) A brief overview of Hadoop 's join method
H joop commonly used jion are reduce side join, map side join, SemiJoin, but reduce side join and map side join are more common, but they are time consuming.
3-14 ) A brief overview of the difference between hadoop 's combinnet and partition
Both combine and partition are functions, and the middle step should only be shuffle ! combine into map ends and reduce end effect is the same key value pairs are combined , can be customized , Partition is the segmentation map results for each node, in accordance with the key mapped to different reduce , but also can customize Defined. In fact, classification can be understood here.
3-15) HDFS data compression algorithm
H has a lot of compression algorithms adoop, which is more commonly used gzip and bzip2 algorithm algorithm, it can be by CompressionCodec be achieved
3-16 ) hadoop scheduling
H adoop scheduling has three of the default scheduling of fifo hadoop. This method is performed according to the priority of the job and the arrival time. There is also a fair scheduler: the meaning of the name means that the user is assigned a fair access to sharing Cluster 呗! Capacity scheduler: the ability to make programs available for execution and obtain resources in the queue.
3-17 ) How much data is output after reduce ?
The amount of output data is not dependent on map end to his data amount, data not reduce also not operational ah !
3-18) Under what circumstances will the datanode not be backed up?
If the three copies saved by Hadoop are not counted as backups, they will not be backed up under normal operating conditions, that is , they will not be backed up when the copy is set to 1. To put it plainly, it is a single machine! !! There is also no backup of the datanode during a forced shutdown or abnormal power failure.
3-19 ) In which process did combine appear?
Hadoop 's map process, do you know the meaning of combining according to the meaning, you will understand the rest. Think about wordcound
3-20) HDFS architecture?
HDFS consists of namenode , secondraynamenode , and datanode .
namenode is responsible for managing datanode and record metadata
secondraynamenode is responsible for merging logs
datanode is responsible for storing data
3-21) The process of hadoop flush ?
Flush is to drop the data to disk and save the data !
3-22) What is a queue
The implementation of the queue is a linked list, and the order of consumption is FIFO.
3-23 ) Three datanodes , what happens when there is an error in one datanode ?
The first will not affect the storage, because other copies are kept, but it is recommended to repair as soon as possible. The second will affect the efficiency of the operation. There are fewer machines. Reduce reduces the number of choices when saving data. One block of data is It's too big so it will be slow.
3-24 ) MapReduce execution process
First, the map end will receive the data from the Text . The text can operate on the data. Finally, the key and value are written to the next step for calculation through the context . The general value received by the reduce is a set that can be calculated. Finally, the data is passed through the context Persist.
From 3 to 25 ) Cloudera which provides several mounting CDH method
· Cloudera Manager
· Tarball
· Yum
· Rpm
3-26 ) Multiple choice and judgment
http://blog.csdn.net/jiangheng0535/article/details/16800415
3-27 ) Hadoop 's combinnet and partition renderings
3-28 ) Hadoop 's rack awareness (or extension)
See picture
The data block is preferentially stored on the machine that is located near the namenode or the machine that is close to the namenode rack. It is just to verify that the sentence does not go to the network if it does not go to the network.
3-29 ) The default file size is 64M , what is the impact of changing to 128M ?
This reduces the processing capacity of the namenode . The metadata of the data is stored on the namenode . If the network is not good, it will increase the storage speed of the datanode . You can set the size according to your network.
3-30 ) When the datanode first joined the cluster , if the log reports incompatible file versions, the format operation is required for the namenode . What is the reason for this?
This is unreasonable, because the namenode format operation is to format the file system.
Technology, the NameNode empty when formatting dfs / name all files in two directories empty, then, will be in the head
Record dfs.name.dir create the file.
Text is not compatible, when it is possible namenode with datanode data in the namespaceID ,
The clusterID is inconsistent. Find the two ID positions and modify them to be the same.
3-31 ) What hadoop streaming ?
Tip: Refers to processing in other languages
3-32 ) In which stages does sorting occur in MapReduce ? Can these sorts be avoided? why?
A MapReduce job consists of two parts: the Map phase and the Reduce phase.
According to sorting, in this sense, the MapReduce framework is essentially a Distributed Sort . In Map
Phase, in the Map phase, the Map Task will output a sort on the local disk by key
Sorting) files (multiple files may be generated in the middle, but will eventually be merged into one), during the Reduce phase, each
Each Reduce Task sorts the received data, so that the data is divided into groups according to the Key , and then
The group is given to reduce () for processing. Many people's misunderstanding is in the Map stage, if you do not use Combiner
There will be no sorting, which is wrong. Whether you use the Combiner or not , the Map Task will sort the generated data.
Ordering (if there is no Reduce Task , there will be no ordering, in fact the ordering in the Map phase is to reduce the Reduce
End sequencing load). Because these sorts are done automatically by MapReduce , the user has no control over them.
hadoop 1.x can not be avoided, it can not shut down, but hadoop2.x is closed.
3-33 ) The concept of hadoop 's shuffer
Shuffer is a process, it is map -to reduce in tune reduce call before data shuffer, mainly partition and sorting, which is the internal cache partition and sub-distribution (that reduce to pull data) and transmission
3-34 ) hadoop optimization
1.The optimization idea can be optimized from the configuration file and system and code design ideas
2. Optimization of the configuration file: adjust the appropriate parameters, and test when adjusting the parameters
3. Optimization of code: The number of combiners should be the same as the number of reduce , and the type of data should be the same, which can reduce the progress of unpacking and encapsulation.
4. System optimization: You can set the maximum number of files opened by the linux system and configure the bandwidth MTU of the network.
5, for the job to add a Combiner is , can greatly reduce the shuffer stage maoTask copied to a remote reduce task data amount, generally combiner and reduce same.
6, as far as possible in the development stringBuffer instead String , String pattern is read-only , and modify it if, be temporary objects, two stringBuffer are modifiable, no temporary object.
7. Modify the configuration:
The following is to modify the mapred-site.xml file
Modify the maximum number of slots
Is the number of slots in each tasktracker on mapred-site.xml provided, the default is 2
<property>
<name> mapred.tasktracker.map.tasks.maximum </ name>
maximum number of tasks
<value> 2 </ value>
</ property>
<property>
<name> mapred.tasktracker.reduce.tasks.maximum </ name>
the maximum number of ducetask
<value> 2 </ value>
</ property>
Adjusting the heartbeat interval
When the cluster size is less than 300 , the heartbeat interval is 300 ms
mapreduce.jobtracker.heartbeat.interval.min heartbeat time
1st Floor, Jinyanlong Office Building, West Building Road, Building Materials City, Changping District, Beijing Tel: 400-618-9090
mapred.heartbeats.in.second For each additional node in the cluster, the time increases by the following value
mapreduce.jobtracker.heartbeat.scaling.factor How much heartbeat increases each time the cluster increases the number above
Start out-of-band heartbeat
mapreduce.tasktracker.outofband.heartbeat is false by default
Configure multiple disks
mapreduce.local.dir
Configure the number of RPC handers
mapred.job.tracker.handler.count is 10 by default and can be changed to 50 , depending on the capabilities of the machine
Configure the number of HTTP threads
tasktracker.http.threads defaults to 40 and can be changed to 100 depending on the capabilities of the machine
Choose the right compression method
Take snappy as an example:
<property>
<name> mapred.compress.map.output </ name>
<value> true </ value>
</ property>
<property>
<name> mapred.map.output.compression.codec </ name>
<value> org.apache.hadoop.io.compress.SnappyCodec </ value>
</ property>
3-35) 3 months datanode have one datanode error will happen?
The data of this datanode will be backed up on other datanodes .
3-36 ) How to determine the number of maps and reduce in mapreduce
In mapreduce , the map is determined by the block size. The number of reduce can be configured according to the user's business.
3-37 ) The problem of merging two files
Given two files a and b , each storing 5 billion urls , each url occupies 64 bytes, the memory limit is 4G , how to find the common urls of a and b files ?
The main idea is to separate the files for calculation, and compare each file to get the same URL. Because the above contains the same URL, there is no need to consider data skew. The detailed problem solving ideas are:
It can be estimated that the size of each file is 5G * 64 = 300G , which is much larger than 4G . So it is impossible to load it completely into memory for processing. Consider a divide-and-conquer approach. Traverse the file a , find hash (url)% 1000 for each url , and then store the urls into 1000 small files (set to a0, a1, ... a999 ) according to the obtained values . So the size of each small file is about 300M . Traverse the file B , and to take a same method url are stored in the 1000 small files (b0, b1 .... b999) of. After processing in this way, all the URLs that may be the same are in the corresponding small files (a0 vs b0, a1 vs b1 .... a999 vs b999) , and small files that do not correspond (such as a0 vs b99 ) cannot have the same URL . Then we only require 1000 pairs of small files with the same URL
Just fine. For example, for a0 vs b0 , we can traverse a0 and store the urls in hash_map . Then traverse B0 , if url in hash_map in, then this url in a and b exist in, can be saved to a file. If the divided small files are uneven and some small files are too large (for example, larger than 2G ), you can consider dividing these too small small files into small files in a similar way
3-38 ) How to determine the number of map and reduce for a job
number of map are generally determined by the DFS block size hadoop cluster, i.e. the total number of blocks of the input file , the reduce side copying map data terminal, with respect to map tasks end, the reduce node resources are compared to the comparative lack of run the speed will slow down, the number for the task should be 0.95 lived 1.75 .
3-39 ) Hadoop 's sequencefile format, and explain what is JAVA serialization, and how to achieve JAVA serialization
1.hadoop 's serialization ( sequencefile) is stored in a binary form
2.Java serialization is about streaming the content of objects
3, implement the serialization need to implement Serializable interfaces will be able to
3-40 ) A brief overview of the differences between hadoop1 and hadoop2
The biggest difference between Hadoop2 and hadoop1 is that the HDFS architecture and mapreduce are greatly different, and the speed has been greatly improved. The two main changes of hadoop2 are: the namenode can be deployed in a cluster, and the jobTracker in mapreduce in hadoop2 The resource scheduler and lifecycle management are split into two independent components and named YARN
3-41 ) New features of YARN
YARN is hadoop2.x only after the mainly hadoop of HA ( ie clusters ) , disk fault tolerance, resource scheduler
3-42 ) The principle of hadoop join
To implement the join of two tables, you first need to mark the table on the map side, label one of the tables, and perform the Cartesian product operation on the reduce side, which is the actual link operation performed by reduce .
3-43 ) Hadoop 's secondary sort
Hadoop defaults to HashPartitioner sorting. When a file on the map side is very large and the other file is very small, the allocation of resources is uneven. You can use setPartitionerClass to set the partition, which forms a secondary partition.
3-44 ) How many stages of hadoop 's mapreduce ordering occur?
Occurs in two phases, even in the map and reduce phases
3-45 ) Please describe the workflow of the shuffle stage in mapreduce . How to optimize the shuffle stage?
Mapreduce 's shuffer is in the process from map task to reduce task . First, it will enter the copy process. It will request the task tracker where the map task is located to obtain the output file of the map task through the http method . Therefore, when the map task ends, these it will fall to the disk file, merge it map the end of the action, but the map copied value, will put the memory buffer, to shuffer use, the reduce phase, constantly merge the files on the disk will eventually.
3-46 ) What is the role of mapreduce 's combiner and when is it not easy to use? ?
The Combiner in Mapreduce is set to avoid data transmission between map tasks and reduce tasks. Hadoop allows users to specify a merge function for the output of map tasks . In order to reduce the amount of data transmitted to Reduce . It is mainly to reduce the output of the Mapper to reduce the network bandwidth and the load on the Reducer .
It should not be used when the amount of data is small.
Zookeeper related
4-1 ) to write you zookeeper understandingWith the rapid development of big data and the coordination of multiple machines to avoid the single point of failure of the main machine, a software to manage the machine is introduced. It is zookeeper to assist the normal operation of the machine.
The Z ookeeper There are two roles leader and follower, wherein the master node is the leader, the other is a slave node, the mounting configuration must pay attention to an odd number of machine configuration, to facilitate fast switching zookeeper election other machines.
When other software executes tasks, when registering with zookeeper, a corresponding directory will be generated under zookeeper so that zookeeper can manage the machine.
4-2 ) The construction process of zookeeper
The main configuration file zoo.cfg configuration dataDir path of a dataLogDir path and myid configuration and the server configuration, the heartbeat port and port election
Hive related
5-1 ) How does hive save metadataThe ways to save metadata are: memory database rerdy , local mysql database, remote mysql database, but local mysql data is used more because local read and write speeds are faster
5-2 ) The difference between external tables and internal tables
First, let's talk about the differences between internal tables and external tables in Hive:
When Hive creates an internal table, it will move the data to the path pointed to by the data warehouse. If an external table is created, only the path where the data is located will be recorded, and no changes will be made to the location of the data. When a table is deleted, the metadata and data of the internal table are deleted together, while the external table only deletes the metadata, not the data. In this way, the external table is relatively more secure, the data organization is more flexible, and it is convenient to share the source data.
5-3 ) For hive , what UDF functions have you written and what is the role
UDF: Abbreviation for user defined function , two ways to write hive udf extend UDF rewrite evaluate the second extend GenericUDF rewrite initialize , getDisplayString , evaluate method
5-4) Hive of sort by and order by the difference between
order by will globally sort the input, so there is only one reducer (multiple reducers cannot guarantee global ordering) and only one reducer, which will cause longer calculation time when the input scale is large.
sort by is not a global sort, it completes the sort before the data enters the reducer.
Therefore, if you use sort by to sort and set mapred.reduce.tasks> 1, then sort by only guarantees the order of each reducer's output, and does not guarantee global order.
5-5 ) How does hive store metadata and what are its characteristics?
1. Hive has a memory database derby database, which is characterized by small data storage and instability
2, mysql database, storage mode can be set by yourself, and persistence is good, general enterprise development uses mysql as a support
5-6 ) What suggestions for using external tables in development?
1.The external table will not be loaded into hive. Only one reference will be added to the metadata.
2, will not delete the table when deleting, only delete the metadata, so do not worry about the data
5-7 ) Hive Partition partition
Partitioned table, dynamic partitioning
5-8 ) What is the difference between insert into and override write ?
insert into : write data from one table to another
override write : Overwrite previous content.
Hbase related
6-1 ) How is HBase 's rowkey created better? How is it better to create column families?Rowkey is a binary code stream. The length of Rowkey has been suggested by many developers to be designed in 10 ~ 100 bytes, but it is recommended to keep it as short as possible. Having an index when searching will speed things up.
Rowkey hash principle , Rowkey only principle , for transaction data Rowkey design , for statistics Rowkey design , for general-purpose data Rowkey design , multi-criteria queries RowKey design .
Summary design column family:
1.It is generally not recommended to design multiple column families
2.Design of data block cache
3.Radical cache design
4, the Bloom filter design ( can improve the speed of random read )
5.Design of production date
6.Column family compression
Unit time version
6-2 ) Implementation of Hbase
Hbase 's implementation principle is the rpc Protocol
6-3) hbase filter implementation principles
I think there is a problem with this problem. There are many filters. I do n’t know which one !!!!
Hbase filter has: the RowFilter , PrefixFilter , KeyOnlyFilter , RandomRowFilter , InclusiveStopFilter , FirstKeyOnlyFilter , ColumnPrefixFilter , ValueFilter , ColumnCountGetFilter , SingleColumnValueFilter , SingleColumnValueExcludeFilter , WhileMatchFilter , FilterList
You see so many filters, who knows the one you asked! !!
The more commonly used filters are: RowFilter A filter that knows that it is a row to filter the information of the row. PrefixFilter prefix filter is to use the prefix as a parameter to find data! The rest is OK without explaining the direct meaning of the filter . It's simple.
6-4 ) Describe HBase, zookeeper setup process
Zookeeper problem upstairs climb step by step, HBase main configuration file has hbase.env.sh main configuration is JDK path and whether to use an external ZK , HBase-the site.xml main configuration with HDFS path of the link And zk information, modify the configuration of other servers linked to regionservers.
6-5 ) How to tune hive ?
When optimizing pay attention to the problem of data, minimizing data skew problem, reduce job number of small files colleagues conducted into a large file, if optimized design so much the better, because the hive computing is mapReduce so adjust The mapreduce parameters also improve performance, such as adjusting the number of tasks .
6-6 ) Setting of hive permissions
Hive permissions need to be set in the hive-site.xml file to take effect. The default configuration is false. You need to set hive.security.authorization.enabled to true and set different permissions for different users, such as select. Operations such as drop.
6-7) hbase write data principle
1. First, the client requests the address of the target data by accessing the ZK.
2. The address of the -ROOT- table is stored in ZK, so ZK requests the data address by accessing the -ROOT- table.
3. Similarly, the -ROOT- table stores the information of .META. You can obtain the specific RS by accessing the .META. Table.
4. .META. Table returns specific RS address to Client after querying specific RS information.
5. After the client obtains the target address, it sends a data request directly to the address.
6-8 ) What to do if hbase is down?
After HBase 's RegionServer is down for more than a certain period of time, HMaster will redistribute its managed regions to other active RegionServers . Since data and logs are persistent in HDFS , this operation will not cause data loss. So data consistency and security are guaranteed. However, the reallocated region needs to restore the memory MemoryStore table in the original RegionServer according to the log . This will cause the down region to be unable to provide external services during this time. Once redistributed, the down node is equivalent to a new RegionServer joining the cluster. In order to balance, some regions need to be distributed to the server again . Therefore, how to make the memory table memstore of the Region Server more available between nodes is a big challenge for HBase .
6-9 ) Hbase in metastore used to do?
Hbase of metastore is used to save data, where the data is saved, there are three methods are the first to the second is the local store, the second is a remote storage that an enterprise with more
6-10) How to optimize the hbase client on the client?
Hbase uses JAVA to calculate, and the optimization of indexing Java is also applicable to hbase . When using filters, remember to enable bloomfilter, which can improve performance by 3-4 times. Set HBASE_HEAPSIZE to a larger value.
6-11 ) How is hbase pre-partitioned?
How to do pre-partitioning, you can use the following three steps: 1. Sampling, first randomly generate a certain number of rowkeys, and sort the sampled data into a set in ascending order. 2. According to the number of regions in the pre-partition, divide the entire set evenly. , Which is the relevant splitKeys. 3. HBaseAdmin.createTable (HTableDescriptor tableDescriptor, byte [] [] splitkeys) can specify the splitKey for the pre-partition, which is the critical rowkey value between regions
6-12 ) How to import mysql data into hbase ?
不能使用 sqoop，速度太慢了，提示如下：
A、一种可以加快批量写入速度的方法是通过预先创建一些空的 regions，这样当
数据写入 HBase 时，会按照 region 分区情况，在集群内做数据的负载均衡。
B、hbase 里面有这样一个 hfileoutputformat 类，他的实现可以将数据转换成 hfile
格式，通过 new 一个这个类，进行相关配置,这样会在 hdfs 下面产生一个文件，这个
时候利用 hbase 提供的 jruby 的 loadtable.rb 脚本就可以进行批量导入。
6-13）谈谈 HBase 集群安装注意事项？
需要注意的地方是 ZooKeeper 的配置。这与 hbase-env.sh 文件相关，文件中
HBASE_MANAGES_ZK 环境变量用来设置是使用 hbase 默认自带的 Zookeeper 还
是使用独立的 ZooKeeper。HBASE_MANAGES_ZK=false 时使用独立的，为 true 时
使用默认自带的。
某个节点的 HRegionServer 启动失败，这是由于这 3 个节点的系统时间不一致相
差超过集群的检查时间 30s。
6-14)简述 HBase 的瓶颈
Hbase主要的瓶颈就是传输问题，在操作时大部分的操作都是需要对磁盘操作的
6-15）Redis, 传统数据库,hbase,hive 每个之间的区别
Redis 是基于内存的数据库，注重实用内存的计算，hbase是列式数据库，无法创建主键，地从是基于HDFS的，每一行可以保存很多的列，hive是数据的仓库，是为了减轻mapreduce而设计的，不是数据库是用来与红薯做交互的。
6-16）Hbase 的特性,以及你怎么去设计 rowkey 和 columnFamily ,怎么去建一个 table
Because hbase is a columnar database and the columns are not part of the table schema , you only need to consider rowkey and columnFamily . Rowkey has a relevant correlation. It is best to add a prefix to the database. The smaller the file, the faster the query speed, and then design the column. There is a column cluster, but it should not be too many.
6-17 ) The difference between Hhase and hive
Apache HBase is a key / value system that runs on HDFS . And Hive is not the same, Hbase can run on its database in real time, instead of running MapReduce tasks. Hive is partitioned into tables, and the tables are further divided into column clusters. Column clusters must be defined using a schema . Column clusters aggregate a certain type of column (columns do not require a schema definition). For example, the " message " column cluster may contain: " to " , " from ", " date " , " subject " , and " body " . Each key / value pair is defined as a cell in Hbase , and each key consists ofrow-key , column cluster, column, and timestamp. In Hbase , a row is a collection of key / value mappings, and this mapping is uniquely identified by row-key . Hbase utilizes Hadoop 's infrastructure and can use general-purpose equipment for horizontal expansion.
6-18 ) Describe the scan and get functions of hbase and their similarities and differences
HBase query implementation only provides two ways: 1. Get the only record by the specified RowKey , get method ( org.apache.hadoop.hbase.client.Get ) 2. Get a batch of records according to the specified conditions, scan method ( org .apache.hadoop.hbase.client.Scan ) The scan method is used to implement the condition query function
6-19 ) The difference between HBase scan setBatch and setCaching
can can increase the speed (set space for time) through setCaching and setBatch methods,
The value setCaching sets is the number of request records per rpc. The default value is 1. The cache can be large to optimize performance, but it will take a long time to transmit once it is too large.
setBatch sets the column size to be fetched each time; some rows are particularly large, so they need to be passed to the client separately, that is, several columns of one row at a time.
6-20 ) Structure of cell in hbase
The data in the cell is of no type, and all are stored in bytecode form.
6-21 ) Conflicts caused by too many regions and too large regions in hbase
H Base of the region will automatically Split , when the region while too, Regio much time distribution will be uneven , while for large quantities of data are substituted into the following recommendations:
1, or must let the business side of rowkey pre-fragmented traffic data rowkey be md5 or other hash strategy so that randomly distributed data as much as possible rather than sequential writes.
2. Observe the size of the region at any time to see if a large region appears .
Flume related
7-1 ) Flume does not collect Nginx logs, but logs via Logger4j . What are the advantages and disadvantages?When nginx collects logs, session information cannot be obtained . However, logger4j can obtain session information. The logger4j method is relatively stable and does not cause downtime. Disadvantages: Not flexible enough. The logger4j method is tightly combined with the filtering of the project. The two flume method is more flexible, easy to plug and unplug, and will not affect project performance.
7-2 ) The difference between flume and kafka collection logs, when the collection of logs stopped in the middle, how to record the previous logs.
Flume collects logs by collecting logs directly to the storage layer through streaming, and kafka tries to talk about log cache in kafka
Cluster, which can be collected to the storage layer later. Flume stopped in the middle of the collection, and the previous log can be recorded in a file manner, while Kafka records the previous log in an offset ( offset ) manner.
Kafka related
8-1 ) Kafka used to live in how data is stored, yo and structure, the Data ..... how many directory partitions, each partition storage format is what?1 , topic is stored according to "topic name - partition"
2.The number of partitions is determined by the configuration file
3. The two most important files under each partition are 0000000000.log and 000000.index , 0000000.log
Rollback at the default 1G size.
Spark related
9-1 ) Difference between mr and spark , how to understand spark-rddMr is a file-based distributed computing framework that records intermediate results and final results in files, and map and reduce data distribution are also in files.
Spark is an iterative computing framework for memory. The intermediate results of the calculation can be cached in memory or cached on the hard disk.
Spark-rdd is a partition of data record set, is calculated using the memory, Spark fast because there are pattern memory
9-2 ) a brief description spark of wordCount the process of implementation
scala> sc.textFile ("/ usr / local / words.txt")
res0: org.apache.spark.rdd.RDD [String] = /usr/local/words.txt MapPartitionsRDD [1] at textFile at <console>: 22
scala> sc.textFile ("/ usr / local / words.txt"). flatMap (_. split (""))
res2: org.apache.spark.rdd.RDD [String] = MapPartitionsRDD [4] at flatMap at <console>: 22
scala> sc.textFile ("/ usr / local / words.txt"). flatMap (_. split ("")). map ((_, 1))
res3: org.apache.spark.rdd.RDD [(String, Int)] = MapPartitionsRDD [8] at map at <console>: 22
scala> sc.textFile ("/ usr / local / words.txt"). flatMap (_. split ("")). map ((_, 1)). reduceByKey (_ + _)
res5: org.apache.spark.rdd.RDD [(String, Int)] = ShuffledRDD [17] at reduceByKey at <console>: 22
scala> sc.textFile ("/ usr / local / words.txt"). flatMap (_. split ("")). map ((_, 1)). reduceByKey (_ + _). collect
res6: Array [(String, Int)] = Array ((dageda, 1), (xiaoli, 1), (hellow, 4), (xisdsd, 1), (xiaozhang, 1))
9-3 ) Use Spark to write programs as required
A. The format of the current file a.text is, please count the number of occurrences of each word
A, b, c, d
B, b, f, e
A, a, c, f
sc.textFile ("/ user / local / a.text"). flatMap (_. split (",")). map ((_, 1)). ReduceByKey (_ + _). collect ()
or:
package cn. bigdata
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD
object Demo {
/ *
a, b, c, d
b, b, f, e
a, a, c, f
c, c, a, d
* Count the number of occurrences of each element in the fourth column
* /
def main ( args : Array [ String ]): Unit = {
val conf : SparkConf = new SparkConf (). setAppName ( "demo" ). setMaster ( "local" )
val sc : SparkContext = new SparkContext ( conf )
val data : RDD [ String ] = sc . textFile ( "f: //demo.txt" )
// Data segmentation
val fourthData : RDD [( String , Int)] = data . map { x =>
val arr : Array [ String ] = x . split ( "," )
val fourth : String = arr ( 3 )
( fourth , 1 )
}
val result : RDD [( String , Int)] = fourthData . reduceByKey (_ + _);
println ( result . collect () . toBuffer )
}
}
B , the HDFS there are two files a.text and b.text, the format of the file (ip, username), such as: a.text, b.text
a.text
127.0.0.1 xiaozhang
127.0.0.1 xiaoli
127.0.0.2 wangwu
127.0.0.3 lisi
B.text
127.0.0.4 lixiaolu
127.0.0.5 lisi
Each file has at least 10 million lines. Please use the program to complete the work.
1) The IP of each file
2) appears in b.text but does not appear in a.text of IP
3) for each user the number of occurrences and each user corresponding IP number
code show as below:
1 ) IP number of each file
package en.bigdata
import java.util.concurrent.Executors
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.LocatedFileStatus
import org.apache.hadoop.fs.Path
import org.apache.hadoop.fs.RemoteIterator
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD
import org.apache.spark.rdd.RDD.rddToPairRDDFunctions
// IP number of each file
object Demo2 {
val cachedThreadPool = Executors.newCachedThreadPool ()
def main (args: Array [String]): Unit = {
val conf: SparkConf = new SparkConf (). setAppName ("demo2"). setMaster ("local")
val sc: SparkContext = new SparkContext (conf)
val hdpConf: Configuration = new Configuration
val fs: FileSystem = FileSystem.get (hdpConf)
val listFiles: RemoteIterator [LocatedFileStatus] = fs.listFiles (new Path ("f: // txt / 2 /"), true)
while (listFiles.hasNext) {
val fileStatus = listFiles.next
val pathName = fileStatus.getPath.getName
cachedThreadPool.execute (new Runnable () {
override def run (): Unit = {
println ("=========================" + pathName)
analyseData (pathName, sc)
}
})
}
}
def analyseData (pathName: String, sc: SparkContext): Unit = {
val data: RDD [String] = sc.textFile ("f: // txt / 2 /" + pathName)
val dataArr: RDD [Array [String]] = data.map (_. split (""))
val ipAndOne: RDD [(String, Int)] = dataArr.map (x => {
val ip = x (0)
(ip, 1)
})
val counts: RDD [(String, Int)] = ipAndOne.reduceByKey (_ + _)
val sortedSort: RDD [(String, Int)] = counts.sortBy (_._ 2, false)
sortedSort.saveAsTextFile ("f: // txt / 3 /" + pathName)
}
}
2 ) appears in b.txt but does not appear in a.txt of ip
package en.bigdata
import java.util.concurrent.Executors
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD
/ *
* Appears in b.txt but does not appear in a.txt of ip
* /
object Demo3 {
val cachedThreadPool = Executors.newCachedThreadPool ()
def main (args: Array [String]): Unit = {
val conf = new SparkConf (). setAppName ("Demo3"). setMaster ("local")
val sc = new SparkContext (conf)
val data_a = sc.textFile ("f: //txt/2/a.txt")
val data_b = sc.textFile ("f: //txt/2/b.txt")
val splitArr_a = data_a.map (_. split (""))
val ip_a: RDD [String] = splitArr_a.map (x => x (0))
val splitArr_b = data_b.map (_. split (""))
val ip_b: RDD [String] = splitArr_b.map (x => x (0))
val subRdd: RDD [String] = ip_b.subtract (ip_a)
subRdd.saveAsTextFile ("f: // txt / 4 /")
}
}
3 )
package en.bigdata
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD
import scala.collection.mutable.Set
/ *
* Each user appears times, and each user corresponding ip number
* /
object Demo4 {
def main (args: Array [String]): Unit = {
val conf = new SparkConf (). setAppName ("Demo4"). setMaster ("local")
val sc = new SparkContext (conf)
val data: RDD [String] = sc.textFile ("f: // txt / 5 /")
val lines = data.map (_. split (""))
val userIpOne = lines.map (x => {
val ip = x (0)
val user = x (1)
(user, (ip, 1))
})
val userListIpCount: RDD [(String, (Set [String], Int))] = userIpOne.combineByKey (
x => (Set (x._1), x._2),
(a: (Set [String], Int), b: (String, Int)) => {
(a._1 + b._1, a._2 + b._2)
},
(m: (Set [String], Int), n: (Set [String], Int)) => {
(m._1 ++ n._1, m._2 + n._2)
})
val result: RDD [String] = userListIpCount.map (x => {
x._1 + ": userCount:" + x._2._2 + ", ipCount:" + x._2._1.size
})
println (result.collect (). toBuffer)
}
}
Sqoop related
10-1 ) How does sqoop guarantee duplicate data when importing into MySql database? What should I do if it is repeated? ?
Add the following command as a node after the statement when importing:
--incremental append \
--check-column id \
--last-value 1208
Redis related
10-1 ) Redis save disk time
# Note: you can disable saving at all commenting all the "save" lines.
#
# It is also possible to remove all the previously configured save
# points by adding a save directive with a single empty string argument
# like in the following example:
#
# save ""
save 900 1
save 300 10
save 60 10000
Environment configuration
1 ) How big is your cluster?
This depends on the size of the individual in the company. Here are some configurations of our company:
Lenovo System x3750 server, the price of 3.5 million and the memory capacity of 32G , product type rack, hard disk interface SSD, CPU frequency 2.6GH, CPU number 2 stars, three-level cache 15MB, cpu core 6 -core, the CPU threads 12 threads , maximum The memory supports 1.5T, the network is a Gigabit network card , and the hard disk interface has 12 card slots when pluggable, and the capacity is configured with 1T
Comments
Post a Comment